This week, three Compute Engine VMs in my Google Cloud project were compromised and briefly used for crypto mining—the VMs were hacked.

Google detected the activity automatically, suspended the instances, and sent me a notification. That was an interesting email to receive!

Vibe-coding a database cluster in Google Cloud

Last Sunday, I created three Ubuntu VMs in us-east5 to run a small YugabyteDB cluster. I wanted to test out some things with Yugabyte and also play with cross-cloud connectivity from an AWS application. After the experiment, I shut the VMs down because I was a little suspicious of leaving them running exposed to the internet. A good precaution, clearly.

Later in the week, I restarted them from my phone to demonstrate the setup to someone. Three hours later, CPU usage spiked, and GCP abuse detection sent me a notification. The Cloud Abuse logs flagged CRYPTO_MINING findings on all three instances, and monitoring showed that CPU went through the roof for these three machines.

It's pretty clear what happened—the VMs were internet-reachable and had multiple database/service ports exposed publicly, including Yugabyte. Some script must have detected the open ports, ran an exploit, and installed the crypto mining software.

No harm done—all this is just a sandbox, and I deleted the VMs. To be honest, like I wrote earlier, this is the best way to really experience the pros and cons of agentic coding, and while it is a pretty obvious lesson for any cybersecurity professional, experiencing it so personally still feels different.

I'll have my agentic assistant create a new cluster. This time, better protected!

I have been playing around with my cloud-based "Tulip Broker" application again. I am using this sample application to learn about AI coding and to try some things with cloud databases. It has been great to try this hands-on; this teaches the concept much better than just in the abstract.

Yes, of course I know that a “scan” is just best-effort in DynamoDB, and of course I know I should have created an index—but that is the difference between knowing and experiencing.

I am preparing for performance testing and noticed the UI was not reliably showing the newest orders. It turned out the backend was using a DynamoDB scan and then sorting client-side. Of course, it kind of worked when I only had a handful of orders, but once I exceeded 50 orders, it was not deterministic anymore.

I configured OpenAI Codex to connect to my AWS account (locked down, with minimum permissions), and it helped diagnose the missing GSI. I then had it backfill older rows by writing entity="ORDER" to existing order items using a one-off Python script, and that resolved it.

On a personal note—this continues to be an amazing experience. Unbelievable how much the software engineering field is changing, and awesome to be part of it.

I've successfully migrated the site to a new Drupal version. I've been on Drupal 7 for several years but supported ended a while back, and when I had some time this weekend I worked through the upgrade.

This was the first upgrade where I could use ChatGPT -- it was very helpful.

The site now runs on a new EC2 instance, with new monitoring and automatic scripts. I stopped using Amazon RDS -- it was overkill for which I needed and too expensive. Instead I'm back at running everything on a single EC2 instance, website and database, and it's working well.

Last year I switched over to T-Mobile 5G after months of back-and-forth with Comcast. They simply couldn’t resolve the reliability issues on our street. I was hesitant at first, but the 5G connection has been excellent. And to state the obvious: reliable internet matters a lot to me given the number of video presentations I do. I’m mostly in the office these days, but for video production, my home studio is still hard to beat.

For the six months I kept the 5G gateway in the basement while testing the signal, and once I was confident in its stability, I decided to give it a permanent (and better) home. Two electricians ran Ethernet to several rooms upstairs — a surprising amount of work, two floors up and across the house — but the result is fantastic.

I moved the modem to the upstairs room that’s closest to the T-Mobile antenna a few streets over — and the visual speaks for itself. The signal strength improved by about 15 decibels, a substantial boost that translates to roughly 30X more received signal power.

Our house has been experiencing intermittent internet issues. Our cable provider, Comcast, has come out a few times to take a look but they have not found the root cause yet.

I spent some time creating a dashboard for myself, so I could track the problem better. I installed Docker on a Raspberry Pi, and installed Prometheus, InfluxDB and Grafana plus custom scrapers for my Motorola modem. All this came out quite nicely.

The first chart shows packet loss -- I run a ping test every few seconds against a couple of internet targets. It's very clear that this connection is not clean.

The other charts show details from the modem. For example, when the Comcast engineer worked on my cables yesterday, it resulted in much higher Upstream Power -- which we thought might have resolved it, but sadly didn't.

I'm mostly done with the Natural Language Processing specialization and tonight I was playing around with a fun proof of concept written by Andrej Karpathy.

The proof of concept is a tiny transformer architecture, "GPT mini". Unlike its big GPT-3 and GPT-4 brothers, this tiny model is character based, not word based. The text I used are three short stories by author Nescio -- which, in fact, constitute his complete works, around 200k characters. I trained it for 20,000 iterations.

The source code and results are available on Github. The architecture uses an embedding size of 192, 6 attention heads and stacks 6 transformer blocks. The results in 2.71M parameters and the 20k iterations took around 15 minutes on an NVIDIA A100.

It's really fun to see how the model improves over time. Initially, the generated text is mostly random, but slowly but surely it starts to reproduce sentence structures and realistic looking words. Of course, with such a small corpus of 200k characters, the model will overfit the data and start to reproduce sentences from the source text -- but it's still cool to see.

After the first iteration

eun nOO nnp nepurK eued d$ptOd nK nt a a K t$utnKtaen Kate dt nd t,pten ap ean d, tpen pnr d K e eaKt nnndddette autaKpaateeu a KpK Kae e h$aKpa ha K tnO a n hatptn aae de antaKpnpptpeKdeddnt npoaupta tp ean hK t p Kao Kt ht p pndn tep nernn auKdtd hnO n ept nK o K eKn n Kapttu ndpt do ate tddon er n KtdnO ed$ ha nudK dtKadddaadtdeaadp p nK$pdteeta t out p tn en nOneKtKanuh an opon Kanee pe dp e d epaKteaKKra aadp n o Kedae eaoata e taeat n hpeuedpK p dn haa en eKedn auu K

After 500 iterations:

I, dat de burs een
heef op dat zei, haar mals dacht de zei vrijn en eeschaan. Wiltij haar zoef
ik op er haarmet van en om de stotjes de waaren in voereed, ik
kaneek z'n had over van wilderde doof toed doe hoeden?" Is in daar
wond een gaamen zest zou zoo van voeren, de del den van
al hij me heeme vand van gon draat op z'n voedene verkorgerd diet
koes opte hemt ond zoo ik geschadens wam op het zageend veellen dondje de eer de
dat naar de verral de van geschiller wijd en een datst een een
gendas a

After 1,000 iteration (training loss 1.54447) -- first mention of "Dora"

Is biesteren er zelfde hand
en geven wit zoo gehand. En wat zei ik zijn op nog meer, nieuw, dat wit was vuor. Hij
waren dat niks was maal, als nooiil met ze zoulven.

Maar dan dan tonden heelemalen, toent naar beziefdig heuis, over
dat den schragen. Dora i weg een meer neem. Een zei i een kleet blauwsten
was moest en knapplook best verschij, dan zooals wees in 't benait zinnen. Er wan
met wweet op deek van de leven en klampte over was gedrhouden, ondergal
weer en gelende sische en hoe of meer d

1500: train loss 1.34416 -- you can see that "Bavink" makes his appearance.

No. En toe kan- hij
dan dat zoo ook, dat ze 't, avonds herd over hem op hun trap
heugen, aan de wijlen of te zeggen. En op haar enoedig, vertielde zag z'n verschicht,
maar eenige schider tot zelf nog en dat heel greest om. En de boogen wat
niks meer zag hij met den handel nog waar boven.

En zei Bavink, wat ze zit wij van z'n beetjes heer met
een pijs af, die zien? En wier zag je heer aan weer anders en dat ze
oude hem. Het wezelfsleven viel was zoo veel gaan, dat i nog haar schoof ofdat,
ze wo

iter 3000: train loss 0.89607
Als-i in de steven lager daar gelezen,
met verlangd jaar er boven. Dan was ik er nergens in. Het was een
raam van hoog.

Op een ander hoofd stond er geweest. En toen ik daar goed was een tijd naar
uit een beetje stroomen en de somberhooge schitterenden van een
dichtertje en niet meer op. De overzij had ik gehad overalen,
dan moet Bavink op een meid voor de lantaarn geluiden, zag ze weer
een duidelijk leven zich neem.

Maar met de zaken heel en das 'm niet een dertje heelemaal in een den hand
va

iter 5500: train loss 0.46585
" En middag on-i ook heeft-i een
betrekking van hun mijn sigaren op een papieren, en toen ik boven
jelui de menschen om niet te houden. Ik ben ben de vrouw niet, maar
mijn gedachten zijn geeft tusschen zijn kleeren verdroeg en een wonderlijk
scht wend dan ons over uit den weg te lochten.

Onderwijl schreef i op z'n zomermiddag waren gestapt en Bavink had aan
gehad w. Z'n boekje leven van den groote gekosten voor dragen in 't huis
en van de Waalt soliabolone tafee en er kreeg naar 't water en dan

15500: train loss 0.23958 -- after 15k iterations, the training loss doesn't really decrease anymore.

7 mensche me schel zoo over boven huis bij de puilten? In een net verden. Ik keek
er naar niet bij over. De zon van die monu stilletjes en geen oogenblik gelijk. En
daarna had je dikke Jan ook weer, den toren van Zierikzee, nu in 't
kleine kuisje in
't water te razen. Daarnaast stond m'n troost bestaan. De treurwilg zij in een volksbuurt, waar weer zeiden in 't
alsterdampje en geen oogenblik 't zelfde. Eenen keer hield ze diep
ze niet verlegenheid zooals als i kleinste vasthiel en verlangen over

I've completed the first three courses of the Natural Language Processing specialization at Coursera, and started the fourth one today. I really enjoyed the the courses so far but this fourth course is the best part: learning about Attention.

The basic concept of this was introduced in 2014, three years later the famous "Attention is all you need" paper came out.

With the rise of Large Language Models, I decided to dive into machine learning again and catch up on how these LLMs truly work. I'm following a course by Deeplearning.ai on Natural Language Processing.

I'm at the end of the second course and we're going through Continuous Bag of Words, which was part of the original word2vec algorithm that was so influential. It's really fun. The previous week was about n-grams, which was also nice to program with.

I successfully completed a certification for Azure, the cloud offering from Microsoft.

I took the exam in our basement; it was strange to be on video all the time. But the process was nice and smooth.