Software Architecture – What Is Supple Design? Coding Blocks podcast

—

Well, this is awkward. Coding Blocks is signing out for now, in this episode we’ll talk about what’s happening and why. We have had an amazing run, far better than we ever expected. Also, Joe recommends 50 games, Allen goes for the gold, and Outlaw is totally normal. (And we’re not crying you’re crying!) Thank you for the support over the last 11 (!!!) years. It’s been a wild ride, and the last thing we ever expected when starting a tech podcast was getting to meet so many fantastic people. View the full show notes here: https://www.codingblocks.net/episode242 Tip of the Week UFO 50 is an odd collection of 50 pseudo-retro video games made by a small group of game developers, most notably including Derek Yu of Spelunky. It’s a unique and specific experience that reminds me of spending the night at your friend’s house who had some console gaming system that you’d only ever heard rumors about. The games seem small and simple at first blush, but there is surprising depth. Favorites so far are Kick Club, Avianos, Attactics, and Mortol. ( Steam ) Use JSDoc annotations to make VSCode “understand” your data ( jsdoc.app ) Can you change your password without needing current password? ( askubuntu.com ) Did you know you can use VS Code for interactive rebasing? How to enable VS Code Interactive Editor ( StackOverflow ) GitLens ( marketplace.visualstudio.com )…

1
Things to Know when Considering Multi-Tenant or Multi-Threaded Applications

42 weeks тому

—

It’s time to talk about the types of things you should consider when diving into multi-tenant applications and things to be especially aware of if you attempt to make things faster by going multi-threaded at the same time. Join us as Allen continues is non-winning / non-losing streak, JZ is taken out by a storm, and Outlaw continues to ignore the notes for the intro to the show… News Reviews Spotify – Colum Ferry Upcoming Events Atlanta Dev Con September 7th, 2024 https://www.atldevcon.com/ DevFest Central Florida on September 28th, 2024 https://devfestflorida.com/ Episode Single vs multi-tenanted applications and multi-threaded programming Be very careful about sharing state when doing anything in multi-threaded AND multi-tenanted applications What are some important things to consider when writing multi-tenanted applications – per Claude As mentioned in the episode, Claude’s answers seem to be very database heavy Data isolation: Ensure that each tenant’s data is securely separated and inaccessible to other tenants. Authentication and authorization: Implement robust systems to verify tenant identity and control access to resources. Scalability: Design the architecture to handle growth in both the number of tenants and data volume per tenant. Performance: Optimize database queries and resource allocation to maintain speed as tenant count increases. Customization: Allow for tenant-specific configurations and features without compromising the core codebase. Monitoring and logging: Implement comprehensive logging and monitoring to track usage, errors, and performance per tenant. Maintenance and updates: Design the system to allow for updates and maintenance with minimal disruption to tenants. Data backup and recovery: Implement tenant-specific backup and recovery processes. Compliance and data regulations: Ensure the system can meet various regulatory requirements that may differ by tenant. Tenant onboarding and offboarding: Create efficient processes for adding new tenants and removing old ones. Good feature flagging support? Tracing / OpenTelemetry Can you elaborate on the data isolation? Data isolation is crucial in multi-tenant systems to prevent unauthorized access and ensure data privacy. Here are some key strategies to ensure data isolation: Separate databases Use a separate database for each tenant Pros: Strong isolation, easier compliance Cons: Higher maintenance, potentially higher costs Shared database with separate schemas Each tenant gets their own schema within a shared database Pros: Good isolation, more cost-effective than separate databases Cons: Requires careful access control, potential for errors Shared schema with tenant ID Use a single schema but include a tenant ID column in each table Pros: Most cost-effective, easiest to maintain Cons: Highest risk of data leaks, requires vigilant query filtering Row-level security Implement database-level security policies to restrict access based on tenant ID Particularly useful with shared schema approach Encryption Encrypt sensitive data, potentially using tenant-specific keys Adds an extra layer of protection even if other measures fail Connection pooling Use separate connection pools for each tenant to prevent session mixups Query filtering Always include tenant ID in queries, preferably at the database level Use ORMs or query builders that automatically add tenant filters API and application-level checks Validate tenant ID in all API calls and application logic Implement middleware to enforce tenant-specific access Resources / Mentions Salable Postgres product – Citus https://www.citusdata.com/ Envelope encryption if you’re wanting to use tenant specific keys for multi-tenanted encryption – one approach is envelope encryption https://en.wikipedia.org/wiki/Hybrid_cryptosystem#Envelope_encryption https://cloud.google.com/kms/docs/envelope-encryption OWASP multi-tenant or “Cloud Tenant” Isolation https://owasp.org/www-project-cloud-tenant-isolation/ C#’s Parallel.ForEach method – the easy way to do parallel processing in C# https://learn.microsoft.com/en-us/dotnet/api/system.threading.tasks.parallel.foreach?view=net-8.0 Can’t remember why we linked episode 11, but here it is! https://www.codingblocks.net/podcast/episode-11-design-patterns-part-1-you-create-me/ Tips of the Week Speculative fix? If you’re not sure that your fix is going to work, or you can’t reproduce the problem then consider over-communicating it and keeping artifacts such as screenshots of what you attempted. It’s useful for communication, but also for looking back when and if you start second guessing yourself. Shapez 2 is a cool refactoring, automation, and puzzle game. It’s similar to Factorio, but with a heavier emphasis on refactoring and puzzles. https://store.steampowered.com/app/2162800/shapez_2/ Kotlin – measureTimeMillis https://kotlinlang.org/api/latest/jvm/stdlib/kotlin.system/measure-time-millis.html Google skills – learn and get certified in Google Cloud https://www.cloudskillsboost.google Pay for a year’s worth of training for $299 and get $200 towards a certification – over $1,500 value https://www.cloudskillsboost.google/payments/new https://www.cloudskillsboost.google/subscriptions From Ivan Kuchin – Dasel – like JQ except it does JSON, YAML, TOML, XML and CSV with zero runtime dependencies https://github.com/TomWright/dasel Google Cloud products in 4 words or less https://cloud.google.com/blog/topics/developers-practitioners/back-popular-demand-google-cloud-products-4-words-or-less-2022-edition https://googlecloudcheatsheet.withgoogle.com/…

1
Two Water Coolers Walk Into a Bar…

44 weeks тому

—

Grab your headphones because it’s water cooler time! This episode we’re catching up on feedback, putting our skills to the test, and wondering what we’re missing. Plus, Allen’s telling it how it is, Outlaw is putting it all together and Joe is minding the gaps! View the full show notes here: https://www.codingblocks.net/episode240 Reviews Thank you again for taking the time to share your review with us! iTunes: Yesso95 Spotify: Auxk0rd, artonus News Atlanta Dev Con September 7th, 2024 https://www.atldevcon.com/ DevFest Central Florida September 28th, 2024 https://devfestflorida.com/ Two water coolers walk into a bar… Several folks share their origin stories in the Coding Blocks slack – especially in episode-discussion Example of dealing with legacy code / hiring people that will work on it ( Episode 239 ) Intentional architecture…what’s the worst that could happen? What’s the sentiment like on Hacker News? ( outerbounds.com ) Cat8 is not small! Why isn’t anything easy? Kubernetes trivia, where are your blind spots? ( proprofs.com ) Ask Claude: Can you give me an example of the kinds of competitions that might exist in a humorous version of the Olympics for programmers? Data gathering and parsing – it doesn’t seem to have gotten much better in decades…are we wrong? Tip of the Week 8 Top Docker Tips and Tricks for 2024 ( docker.com ) Have you tried Eartlhy, like Dockerfiles for all of your builds that you can run locally? ( earthly.dev ) Java’s JavaAgent Explained ( bito.ai ) Mirrord is an alternative to Telepresence that makes working with Kubernetes easier ( mirrord.dev ) Kubernetes + Skaffold + Telepresence + K9s = Winning, it’s a great combination of tools that work great together! https://cloud.google.com/kubernetes-engine?hl=en https://skaffold.dev/ https://www.telepresence.io/ https://k9scli.io/…

1
How did We Even Arrive Here?

46 weeks тому

—

We were asked in one of our recent reviews in Apple Podcasts if we would share our career paths and how we got started and how we arrived where we are today. We took some of that info, threw it at Microsoft’s CoPilot / DALL E 3 and that’s the artwork for this episode – you may be a little surprised at how the 3 of us got started as software developers. We also have some fun things to share about recent purchases, tweaks, mistakes made and some information about the CrowdStrike outage that recently occurred. Reviews Thank you again for taking the time to share your review with us! iTunes: Ksed_NJ News Atlanta Dev Con September 7th, 2024 https://www.atldevcon.com/ DevFest Central Florida September 28th, 2024 https://devfestflorida.com/ This Episode Career paths – someone asked that we talk about our career paths https://en.wikipedia.org/wiki/Lou_Pearlman Enabling trace caused a bug that didn’t happen otherwise – in other words, errors in your code (such as in a toString()) could cause unforseen problems when working with other libraries and enabling verbose logging Upgraded the fans in my Omada switch…may have done something stupid – ie. you should never mess aorund inside electrical components when they were powered on else you may risk injury or even frying some expensive equipment The travel router experience has been awesome – we mentioned this thing in a recent episode and it really is awesome Travel Router: https://amzn.to/3W5U8IS Travel Router Case: https://amzn.to/3zufsyU Great topic in the Coding Blocks slack #episode-discussion channel – can you buy your way out of a jam when older developers leave and tribal knowledge is abundant? ie. how much money would it take for you to work on an old jquery application nowadays, or maybe support a v1 MVC .NET application Some craziness with Crowdstrike and several industries coming to a standstill due to a bug Government Accountability Office blog entry on the CrowdStrike issue https://www.gao.gov/blog/crowdstrike-chaos-highlights-key-cyber-vulnerabilities-software-updates#:~:text=Earlier this month%2C a software,Disruptions were widespread Great explanation of what happened https://www.youtube.com/watch?v=wAzEJxOo1ts When is Programmer’s Day? International Programmer’s day is January 7th, for no reason I can find. No joke, it depends! It’s the 256 day of the year, which changes with leap years. This year it’s on 9/12. Guess we just need to celebrate all 3! https://www.timeanddate.com/holidays/world/international-programmers-day Resources Mentioned the old TRS-80 Radioshack handheld – here’s a page with some info on these dinosaurs https://oldcomputers.net/~oldcompu/trs80pc1.html Tips of the Week Hollow Knight, amazing created by 3 people (mostly) fantastic game, it’s not as hard or long as I thought it would be. The gameplay is amazing, the music is amazing, the story is hidden, but amazing. Just play it on just about any platform. https://store.steampowered.com/app/367520/Hollow_Knight/ Midinous is a cool way of generating midi visually with nodes, similar to decision trees, that can do cool things with randomness and weighting to manipulate all sorts of midi parameters (velocity, mod and pitch wheels, any parameters that your VSTs support!) https://www.youtube.com/watch?v=rtTWtzWav8I https://store.steampowered.com/app/1727420/Midinous/ If that’s not low-level enough for you, check out overtone. It’s an open-source LISP-like wrapper around the SuperCollider audio engine that you can use to make really cool, live looping music. https://overtone.github.io/ Did you know that k9s has a slick way of supporting skins via XDG_CONFIG, you just drop in a couple skin files and then update a parameter to tell your context/clusters what to use. Great article that walks you through it: https://k9scli.io/topics/skins/ Starship minimalist prompt for any shell, previously Joe had been using powerlevel10k to show git information, kubernetes information, auto-complete etc, but it’s been deprecated and it looks like Starship appears to be the de jour replacement. https://starship.rs/ Showing the “hidden frames” in a strack trace in IntelliJ If you want to see everything that happened in those hidden frames, you will want to right click that stack trace and show traces from external libraries – there are other features there you can enable/disable as well! Get outside … go for a hike, a ride, go see a concert, go see Deadpool & Wolverine … That one means your back inside, but you had to go outside to get there Students – get an internship or co-op job … it’s easier to get started in your career while you’re still in school than it will be when you’re out and looking for a job…

1
AI, Blank Pages, and Client Libraries…oh my!

50 weeks тому

—

It’s Water Cooler Time! We’ve got a variety of topics today, and also Outlaw’s lawyering up, Allen can read QR codes now, and Joe is looking at second careers. View the full show notes here: https://www.codingblocks.net/episode238 News As always, thank you for leaving us a review – we really appreciate them! Almazkun, vassilbakalov, DzikijSver Atlanta Dev Con September 7th, 2024 https://www.atldevcon.com/ DevFest Central Florida on September 28th, 2024 Interested? Submit your talk proposal here: https://sessionize.com/devfest-florida-orlando-2024/ Water Cooler How many programmers are there now? ( statista.com ) Are we still growing? What will it be like when we stop growing? What will people be doing instead? AI music generators are being sued! ( msn.com ) Curse of the Blank Page Naming things is important, gives them power…but also the power to defeat them! Don’t make any one specific technology your hammer Client libraries that completely change with server upgrades What’s the most important or relevant thing to learn as a developer now? Do you research or learn on vacation? Tip of the Week Curated, High-Quality Stories, Essays, Editorials, and Podcasts based around Software Engineering. It’s more polished and less experimental than PagedOut ( Github ) Also, there’s a new Paged Out, complete with downloadable art. It’s more avant-garde than GIthub’s Readme project, featuring articles on Art, Cryptography, Demoscenes, and Reverse Engineering. ( pagedout.institute ) Travel Router – Extensible Authentication Protocol (EAP) is used to pass the authentication information between the supplicant (the Wi-Fi workstation) and the authentication server (Microsoft IAS or other) ( Amazon ) Comparison of Travel Routers ( gi.inet.com ) Carrying case for router ( Amazon ) Travel power cube – 6 power outlets followed by 3 ( Amazon ) Did you know you that Windows has a built in camera QR code reader? Guava caching libraries in Java ( Github ) Caffiene is a more recent alternatitive ( Github ) Generative AI for beginners – “Learn the fundamentals of building Generative AI applications with our 18-lesson comprehensive course by Microsoft Cloud Advocates.” Microsoft has a course for getting into generative AI! ( microsoft.github.io ) Claude is better than Chat GPT? ( claude.ai ) How to Get the Most out of Postgres Memory Settings – thanks Mikerg! ( temb.io )…

1
Alternatives to Administering and Running Apache Kafka

52 weeks тому

—

In the past couple of episodes, we’d gone over what Apache Kafka is and along the way we mentioned some of the pains of managing and running Kafka clusters on your own. In this episode, we discuss some of the ways you can offload those responsibilities and focus on writing streaming applications. Along the way, Joe does a mighty fine fill-in for proper noun pronunciation and Allen does a southern auctioneer-style speed talk. View the full show notes here: https://www.codingblocks.net/episode237 Reviews As always, thank you for leaving us a review – we really do appreciate them! From iTunes: Abucr7 Upcoming Events Atlanta Dev Con September 7th, 2024 https://www.atldevcon.com/ DevFest Central Florida on September 28th, 2024 Interested? Submit your talk proposal here: https://sessionize.com/devfest-florida-orlando-2024/ Kafka Compatible and Kafka Functional Alternatives Why? Because running any type of infrastructure requires time, knowledge, and blood, sweat and tears Confluent https://www.confluent.io/confluent-cloud/pricing/ We’ve personally had good experiences with their Kafka as a service WarpStream https://www.warpstream.com/ “WarpStream is an Apache Kafka® compatible data streaming platform built directly on top of object storage: no inter-AZ bandwidth costs, no disks to manage, and infinitely scalable, all within your VPC” ZERO disks to manage 10x cheaper than running Kafka Agents stream data directly to and from object storage with no buffering on local disks and no data tiering. Create new serverless “Virtual Clusters” in our control plane instantly Support different environments, teams, or projects without managing any dedicated infrastructure Things you won’t have to do with WarpStream Upscale a cluster that is about to run out of space Figure out how to restore quorum in a Zookeeper cluster or Raft consensus group Rebalance partitions in a cluster “WarpStream is protocol compatible with Apache Kafka®, so you can keep using all your favorite tools and software. No need to rewrite your application or use a proprietary SDK. Just change the URL in your favorite Kafka client library and start streaming!” Never again have to choose between reliability and your budget. WarpStream costs the same regardless of whether you run your workloads in a single availability zone, or distributed across multiple WarpStream’s unique cloud native architecture was designed from the ground up around the cheapest and most durable storage available in the cloud: commodity object storage WarpStream agents use object storage as the storage layer and the network layer, side-stepping interzone bandwidth costs entirely Can be run in BYOC (bring your own cloud) or in Serverless BYOC – you provide all the compute and storage – the only thing that WarpStream provides is the control plane Data never leaves your environment Serverless – fully managed by WarpStream in AWS – will automatically scale for you even down to nothing! Can run in AWS, GCP and Azure Agents are also S3 compatible so can run with S3 compatible storage such as Minio and others RedPanda Redpanda is a slimmed down native Kafka protocol compliant drop-in replacement for Kafka There’s even a Redpanda Connect! It’s main differentiator is performance, it’s cheaper and faster Apache Pulsar Similar to Kafka, but changes the abstraction on storage to allow more flexibility on IO Has a Kafka compliant wrapper for interchangability Simple data offload functionality to S3 or GCS Multi tenancy Geo replication Cloud alternatives Google Cloud – PubSub https://cloud.google.com/pubsub Azure – Event Hubs https://azure.microsoft.com/en-us/products/event-hubs AWS – Kinesis https://aws.amazon.com/kinesis/ Tip of the Week Chord AI is an Android/iOS app that uses AI to figure out the chords for a song. This is really useful if you just want to get the quick jist of a song to play along with. The base version is free, and has a few different integration options (YouTube, Spotify, Apple Music Local Files for me) and it uses your phones microphone and a little AI magic to figure it out. It even shows you how to play the chords on guitar or piano. The free version gets you basic chords, but you can pay $8.99 a month to get more advanced/frequent chords. https://www.chordai.net/ Pandas is nearly as good, if not better than SQL for exploring data https://pandas.pydata.org/ Another tip for displaying in Jupyter notebooks – to HTML() your dataframes to show the full column data https://www.geeksforgeeks.org/how-to-render-pandas-dataframe-as-html-table/ Take photos or video and convert them into 3d models https://lumalabs.ai/luma-api…

1
Nuts and Bolts of Apache Kafka

1 year тому

—

Topics, Partitions, and APIs oh my! This episode we’re getting further into how Apache Kafka works and its use cases. Also, Allen is staying dry, Joe goes for broke, and Michael (eventually) gets on the right page. The full show notes are available on the website at https://www.codingblocks.net/episode236 News Thanks for the reviews! angingjellies and Nick Brooker Please leave us a review! ( /review ) Atlanta Dev Con is coming up, on September 7th, 2024 ( www.atldevcon.com ) Kafka Topics They are partitioned – this means they are distributed (or can be) across multiple Kafka brokers into “buckets” New events written to Kafka are appended to partitions The distribution of data across brokers is what allows Kafka to scale so well as data can be written to and read from many brokers simultaneously Events with the same key are written to the same partition as the original event Kafka guarantees reads of events within a partition are always read in the order that they were written For fault tolerance and high availability, topics can be replicated…even across regions and data centers NOTE: If you’re using a cloud provider, know that this can be very costly as you pay for inbound and outbound traffic across regions and availability zones Typical replication configurations for production setups are 3 replicas Kafka APIS Admin API – used for managing and inspecting topics, brokers, and other Kafka objects Producer API – used to write events to Kafka topics Consumer API – used to read data from Kafka topics Kafka Streams API – the ability to implement stream processing applications/microservices. Some of the key functionality includes functions for transformations, stateful operations like aggregations, joins, windowing, and more In the Kafka streams world, these transformations and aggregations are typically written to other topics (in from one topic, out to one or more other topics) Kafka Connect API – allows for the use of reusable import and export connectors that usually connect external systems. These connectors allow you to gather data from an external system (like a database using CDC) and write that data to Kafka. Then you could have another connector that could push that data to another system OR it could be used for transforming data in your streams application These connectors are referred to as Sources and Sinks in the connector portfolio ( confluent.io ) Source – gets data from an external system and writes it to a Kafka topic Sink – pushes data to an external system from a Kafka topic Use Cases Message queue – usually talking about replacing something like ActiveMQ or RabbitMQ ** Message brokers are often used for responsive types of processing, decoupling systems, etc. – Kafka is usually a great alternative that scales, generally has faster throughput, and offers more functionality Website activity tracking – this was one of the very first use cases for Kafka – the ability to rebuild user actions by recording all the user activities as events How and why Kafka was developed ( LinkedIn ) Typically different activity types would be written to different topics – like web page interactions to one topic and searches to another Metrics – aggregating statistics from distributed applications Log aggregation – some use Kafka for storage of event logs rather than using something like HDFS or a file server or cloud storage – but why? Because using Kafka for the event storage abstracts away the events from the files Stream processing – taking events in and further enriching those events and publishing them to new topics Event sourcing – using Kafka to store state changes from an application that are used to replay the current state of an object or system Commit log – using Kafka as an external commit log is a way for synchronizing data between distributed systems, or help rebuild the state in a failed system {"@context":"http:\/\/schema.org\/","@id":"https:\/\/www.codingblocks.net\/podcast\/nuts-and-bolts-of-apache-kafka\/#arve-youtube-iuudru9-hrk67f316ae1764d581944153","type":"VideoObject","embedURL":"https:\/\/www.youtube-nocookie.com\/embed\/IuUDRU9-HRk?feature=oembed&iv_load_policy=3&modestbranding=1&rel=0&autohide=1&playsinline=0&autoplay=0"} Tip of the Week Rémi Gallego is a music producer who makes music under a variety of names like The Algorithm and Boucle Infini, almost all of it is instrumental Synthwave with a hard-rock edge. They also make a lot of video game music, including 2 of my favorite game soundtracks of all time “The Last Spell” and “Hell is for Demons” ( YouTube ) Did you know that the Kubernetes-focused TUI we’ve raved about before can be used to look up information about other things as well, like :helm and :events. Events is particularly useful for figuring out mysteries. You can see all the “resources” available to you with “?”. You might be surprised at everything you see (pop-eye, x-ray, and monitoring) WarpStream is an S3 backed, API compliant Kafka Alternative. Thanks MikeRg! ( warpstream.com ) Cloudflare’s trillion message Kafka setup, thanks Mikerg! ( blog.bytebytego.com ) Want the power and flexibility of jq, but for yaml? Try yq! ( gitbook.io ) Zenith is terminal graphical metrics for your *nix system written in Rust, thanks MikeRg! ( github.com ) 8 Big (O)Notation Every Developer should Know ( medium.com ) Another Git cheat sheet ( wizardzines.com )…

1
Intro to Apache Kafka

1 year тому

—

We finally start talking about Apache Kafka! Also, Allen is getting acquainted with Aesop, Outlaw is killing clusters, and Joe was paying attention in drama class. The full show notes are available on the website at https://www.codingblocks.net/episode235 News Atlanta Dev Con is coming up, on September 7th, 2024 ( www.atldevcon.com ) Intro to Apache Kafka What is it? Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications. Core capabilities High throughput – Deliver messages at network-limited throughput using a cluster of machines with latencies as low as 2ms. Scalable – Scale production clusters up to a thousand brokers, trillions of messages per day, petabytes of data, and hundreds of thousands of partitions. Elastically expand and contract storage and processing Permanent storage – Store streams of data safely in a distributed, durable, fault-tolerant cluster. High availability – Stretch clusters efficiently over availability zones or connect separate clusters across geographic regions. Ecosystem Built-in stream processing – Process streams of events with joins, aggregations, filters, transformations, and more, using event-time and exactly-once processing. Connect to almost anything – Kafka’s out-of-the-box Connect interface integrates with hundreds of event sources and event sinks including Postgres, JMS, Elasticsearch, AWS S3, and more. Client libraries – Read, write, and process streams of events in a vast array of programming languages Large ecosystem of open source tools – Large ecosystem of open source tools: Leverage a vast array of community-driven tooling. Trust and Ease of Use Mission critical – Support mission-critical use cases with guaranteed ordering, zero message loss, and efficient exactly-once processing. Trusted by thousands of organizations – Thousands of organizations use Kafka, from internet giants to car manufacturers to stock exchanges. More than 5 million unique lifetime downloads. Vast user community – Kafka is one of the five most active projects of the Apache Software Foundation, with hundreds of meetups around the world. What is it? Getting data in real-time from event sources like databases, sensors, mobile devices, cloud services, applications, etc. in the form of streams of events. Those events are stored “durably” (in Kafka) for processing, either in real-time or retrospectively, and then routed to various destinations depending on your needs. It’s this continuous flow and processing of data that is known as “streaming data” How can it be used? (some examples) Processing payments and financial transactions in real-time Tracking automobiles and shipments in real time for logistical purposes Capture and analyze sensor data from IoT devices or other equipment To connect and share data from different divisions in a company Apache Kafka as an event streaming platform? It contains three key capabilities that make it a complete streaming platform Can publish and subscribe to streams of events Can store streams of events durably and reliably for as long as necessary (infinitely if you have the storage) To process streams of events in real-time or retrospectively Can be deployed to bare metal, virtual machines or to containers on-prem or in the cloud Can be run self-managed or via various cloud providers as a managed service How does Kafka work? A distributed system that’s composed of servers and clients that communicate using a highly performant TCP protocol Servers Kafka runs as a cluster of one or more servers that can span multiple data centers or cloud regions Brokers – these are a portion of the servers that are the storage layer Kafka Connect – these are servers that constantly import and export data from existing systems in your infrastructure such as relational databases Kafka clusters are highly scalable and fault-tolerant Clients Allows you to write distributed applications that allow to read, write and process streams of events in parallel that are fault-tolerant and scale These clients are available in many programming languages – both the ones provided by the core platform as well as 3rd party clients Concepts Events It’s a record of something that happened – also called a “record” in the documentation Has a key Has a value Has an event timestamp Can have additional metadata Producers and Consumers Producers – these are the client applications that publish/write events to Kafka Consumers – these are the client applications that read/subscribe to events from Kafka Producers and consumers are completely decoupled from each other Topics Events are stored in topics Topics are like folders on a file system – events would be the equivalent of files within that folder Topics are mutli-producer and multi-subscriber There can be zero, one or many producers or subscribers to a topic that write to or read from that topic respectively Unlike many message queuing systems, these events can be read from as many times as necessary because they are not deleted after being consumed Deleting of messages is handled on a per topic configuration that determines how long events are retained Kafka’s performance is not dependent on the amount of data nor the duration of time data is stored, so storing for longer periods is not a problem Resources we Like Why Strimzi moved away from statefulsets ( github.com ) Tip of the Week Flipper Zero is a multi-functional interaction device mixed with a Tamagotchi. It has a variety of IO options built in, RFID, NFC, GPIO, Bluetooth, USB, and a variety of low-voltage pins like you’d see on an Arduino. Using the device upgrades the dolphin, encouraging you to try new things…and it’s all open-source with a vibrant community behind it. ( shop.flipperzero.one ) 7 cool and useful things to do with your Flipper Zero Kafka Tui?! Kaskade is a cool-looking Kafka TUI that has got to be better than using the scripts in the build folder that comes with Kafka. ( github.com/sauljabin/kaskade ) Microstudio is a web-based integrated development environment for making simple games and it’s open source! ( microstudio.dev ) Bing Copilot has a number of useful prompts ( bing.com ) Designer (photos) Vacation Planner Cooking assistant Fitness trainer Sharing metrics between projects in GCP, Azure, and maybe AWS??? GCP (projects): ( cloud.google.com ) Azure (resource groups or subscriptions): ( learn.microsoft.com ) AWS (multiple accounts): ( docs.aws.amazon.com ) Checking wifi in your home – Android Only ( play.google.com ) Powering POE without running cables ( Amazon ) Omada specific – cloud vs local hardware ( Amazon ) How to “shutdown” a Kafka cluster in Kubernetes: kubectl annotate kafka my-kafka-cluster strimzi.io/pause-reconciliation="true" --context=my-context --namespace=my-namespace kubectl delete strimzipodsets my-kafka-cluster --context=my-context --namespace=my-namespace Then to “restart” the cluster: kubectl annotate kafka my-kafka-cluster strimzi.io/pause-reconciliation- --context=my-context --namespace=my-namespace https://github.com/strimzi/proposals/blob/main/031-statefulset-removal.md…

1
StackOverflow AI Disagreements, Kotlin Coroutines and More

1 year тому

—

Joe Zack was on a brief holiday so Allen and Michael took over the helm for an episode. What would a new episode be without a little something regarding AI, some more love for Kotlin, and a number of excellent tips throughout (as well as at the end of) the episode. Reviews iTunes: ivan.kuchin News Atlanta Dev Con September 7th, 2024 https://www.atldevcon.com/ Topics People trying to remove their answers from StackOverflow to not allow OpenAI to use their answers without permission/recognition? https://www.tomshardware.com/tech-industry/artificial-intelligence/stack-overflow-bans-users-en-masse-for-rebelling-against-openai-partnership-users-banned-for-deleting-answers-to-prevent-them-being-used-to-train-chatgpt Obfuscate data dumps with PostgreSQL https://github.com/GreenmaskIO/greenmask/ Kotlin Coroutines https://kotlinlang.org/docs/coroutines-overview.html https://kotlinlang.org/docs/coroutine-context-and-dispatchers.html#dispatchers-and-threads Reminded Outlaw of the Cloudflare Workers we mentioned a while back https://developers.cloudflare.com/workers/ Please leave us a review! https://www.codingblocks.net/review You can control if YouTube keeps track of your history (at least that you can see) 100 Things You Didn’t Know About Kubernetes https://www.devopsinside.com/100-things-you-didnt-know-about-kubernetes-part-1/ Do the IDE AI’s really make you more productive? Random Bits Tesla Las Vegas Loop https://www.lvcva.com/vegas-loop/ What actually happens when you overfill the oil in a vehicle? https://www.youtube.com/watch?v=VaTbfvzNbxQ Fisker Ocean totalled after a $900 door ding…really https://jalopnik.com/fisker-ocean-totaled-over-910-door-ding-after-insurer-1851451187 A Ford Mustang painted with the blackest black paint available https://youtu.be/Ll27OkWuE1g Tip of the Week Docker Blog is pretty excellent https://www.docker.com/blog/ Car Research Car reliability information https://www.truedelta.com/ Actual problems logged with car models by year https://www.carcomplaints.com/ Great search engine for finding cars and more metadata about the listing like how long the car has been listed https://caredge.com/ Utilizing wood sheet goods by utilizing cut lists https://www.opticutter.com/cut-list-optimizer Docker’s chicken-n-egg problem Use a multi-stage Dockerfile where an earlier stage has the tools you need Manually dearmor a PGP public key (Hint: it’s the opposite of: https://superuser.com/questions/764465/how-to-ascii-armor-my-public-key-without-installing-gpg ) Download the file using the server suggested name With wget … --content-disposition https://man7.org/linux/man-pages/man1/wget.1.html Wth curl … -JO -J, –remote-header-name -O, –remote-name https://curl.se/docs/manpage.html#-J…

1
Llama 3 is Here, Spending Time on Environmental Setup and More

1 year тому

—

In this episode Joe introduces us to more security items you should be aware of in the world of CWE’s, Michael bends to the will of Joe and Allen in his favorite portion of the show, and Allen pontificates on the time spent setting up IDE’s and environments. Reviews – Thank You! iTunes: Vlad Bezden, Mom in VA, Make1977 Spotify: chutney3000, Xuraith Upcoming Events Atlanta Dev Con September 7th, 2024 https://www.atldevcon.com/ Topics Open Telemetry The backend matters https://opentelemetry.io/ecosystem/integrations/ Some backends are more fully featured than others Splunk Trace Analyzer https://docs.splunk.com/observability/en/apm/apm-spans-traces/trace-analyzer.html Google Trace Explorer https://cloud.google.com/trace/docs/finding-traces Azure OTel Guide https://learn.microsoft.com/en-us/azure/azure-monitor/app/opentelemetry-enable?tabs=aspnetcore AWS OTel Information https://aws.amazon.com/otel/ The processor can decouple you https://opentelemetry.io/docs/collector/configuration/#processors CNCF – Cloud Native Computing Foundation If you’re working in a cloud environment, you should know the projects here https://www.cncf.io/projects/ Super cool visualization tool for the projects https://landscape.cncf.io/ Llama 3 – the next version of Meta’s AI engine “Now available with both 8B and 70B pretrained and instruction-tuned versions to support a wide range of applications” https://llama.meta.com/llama3/ Environmental concerns over the processing required for AI Power requirements for processing some of the LLM’s https://www.nnlabs.org/power-requirements-of-large-language-models/ The Microsoft underwater datacenter https://news.microsoft.com/source/features/sustainability/project-natick-underwater-datacenter/ Setting up IDE’s and environments IDE vs old school debugging Setup can require a significant amount of time Is it worth it? What if you’re just working on a bug? Security Resources What’s the difference between CWE and OWASP? CWE (Common Weakness Enumeration) is a community-developed list of common software and hardware weaknesses. It’s similar to OWASP, but older (1999 vs 2001) and more general – including non web apps and (more recently) hardware The infamous “NVD” database links CVE (Common Vulnerabilities and Exposures) to CWE https://nvd.nist.gov/vuln/detail/CVE-2021-44228 https://cwe.mitre.org/top25/archive/2023/2023_trends.html Tips Pre-warning – probably wouldn’t recommend installing this! Saw a cool Windows utility called “Windrecorder” that records video and text from your desktop, and lets you rewind and search. Uses ffmpeg to record screen into small 15-minute fragment files Search(by window titles, text keywords, or descriptions of images) Everything happens should only on your computer Cons: No instant rewind (have to be out of the window), Storage is unencrypted, Not much LLM / ML fancy stuff…and security https://tonoko.notion.site/I-made-an-open-source-app-to-rewind-search-everything-happened-on-your-screen-on-Windows-184d1a9d5edb494dba0c2f46d311ec5c https://github.com/yuka-friends/Windrecorder MacOS’s Spotlight is more powerful than you maybe knew https://www.intego.com/mac-security-blog/spotlight-secrets-15-ways-to-use-spotlight-on-your-mac/ https://beebom.com/spotlight-tips-tricks/ If you’re grep command isn’t working like you thought it should, you might be a victim of content getting kicked out of the buffer grep --line-buffered iOS – get text from images https://support.apple.com/guide/iphone/use-live-text-iphcf0b71b0e/ios…

1
Ktor, Logging Ideas, and Plugin Safety

1 year тому

—

Picture, if you will, a nondescript office space, where time seems to stand still as programmers gather around a water cooler. Here, in the twilight of the workday, they exchange eerie tales of programming glitches, security breaches, and asynchronous calls. Welcome to the Programming Zone, where reality blurs and (silent) keystrokes echo in the depths […]…

1
Importance of Data Structures, Bad Documentation and Comments and More

1 year тому

—

In this episode, Allen, Joe and Michael finally make it back to record together! Allen revisits the basics, Michael kicks off boomer hour nicely, and JZ let’s us know that the dream of an 8-bit looking keyboard is not dead. News Topics Tips

1
Decorating your Home Office

1 year тому

—

This time we are missing the “ocks”, but we hope you enjoy this off…ice topic chat about personalizing our workspaces. Also, Joe had to put a quarter in the jar, and Outlaw needs a cookie. The full show notes are available on the website at https://www.codingblocks.net/episode230 News Thank you for the review Szymon! Want to leave […]…

1
Multi-Value, Spatial, and Event Store Databases

1 year тому

—

We are mixing it up on you again, no Outlaw this week, but we can offer you some talk of exotic databases. Also, Joe pronounces everything correctly and Allen leaves you with a riddle. The full show notes are available on the website at https://www.codingblocks.net/episode229 News Multivalue DBMS Spatial DBMS Event Stores Resources we like […]…

1
Overview of Object Oriented, Wide Column, and Vector Databases

1 year тому