How to automate improving our hand-rolled anti-spam captcha using LLama3.2-Vision
Adrian Latham
As we shipped our hand-rolled anti-spam CAPTCHA a few days ago, let's revisit it to see if it's actually any good at disguising the text.
This is what it generates at the moment.
Okay. As I have llama3.2-vision::11b running locally, let's give it our CAPTCHA and find out what it can infer.
ollama run llama3.2-vision:11b >>> Send a message (/? for help)
With Ollama on the command line, I just need to pass it the path to the file:
>>> captcha.png
Added image 'captcha.png'
The information in the question is:
* A mathematical expression is given: 300 + 1
* The task is to evaluate the result of this expression.
## Step 1
First, we need to solve the addition problem.
## Step 2
Adding 1 to 300 gives us 301.
The final answer is: $oxed{301}$
Oops! It solved it immediately. And without any prompting!
Okay..let's add some noise to the CAPTCHA. Say, a random distribution of dots.
const numDots = 100;
for (let i = 0; i < numDots; i++) {
// Generate random coordinates
const x = Math.random() * 200;
const y = Math.random() * 100;
// Set dot color to black
ctx.fillStyle = colours.color;
// Draw a circle (dot) at the random coordinates
ctx.beginPath();
ctx.arc(x, y, 1, 0, 2 * Math.PI); // Radius of 1 pixels
ctx.fill();
}
And let's take a peek.
Let's see how we get on with this CAPTCHA!
>>> captcha.png
Added image ''captcha.png''
The information in the question is:
* A mathematical expression is given: 700 + 5
* The task is to evaluate the result of this expression.
## Step 1
First, we need to solve the addition problem.
## Step 2
Adding 5 to 700 gives us 705.
The final answer is: $oxed{705}$
Okay, okay...why don't we automate generating a CAPTCHA and then piping it to a model to see if it can solve it each time? And perhaps create a loop so that if it can work out the correct answer, we keep adding more and more random dots until it is unable to solve it?
Let's break it down into a few steps:
- Create a script to manage the entire process - let's call it captcha-runner.js
- Update captcha-runner.js so that it pipes the image to the model and asks it to transcribe the image
- Update captcha-runner.js so that it compares the response with the original equation (essentially comparing like for like)
- If the model accurately transcribed the text, we need to add more dots and try again
We'll also need to revisit the captcha.js so that we can set the number of dots when we call it.
{
"dots": 100
}
Let's revisit the original code to make it more tweakable..
captcha.js
// Adding weights parameter...
const generateCaptcha = (weights) => {
// updating loop...
const { dots: numDots } = weights;
for (let i = 0; i < numDots; i++) {
// Generate random coordinates
const x = Math.random() * 200;
const y = Math.random() * 100;
// Set dot color to black
ctx.fillStyle = colours.color;
// Draw a circle (dot) at the random coordinates
ctx.beginPath();
ctx.arc(x, y, 1, 0, 2 * Math.PI); // Radius of 1 pixels
ctx.fill();
}
// ...
The "weights" object is the tweakable bit.
Okay, it's one thing manually calling Ollama from the command line, but if I want to do it programmatically I should use the RESTful API exposed locally.
I needed to remind myself what that looks like using curl before I start calling it in code.
curl -X POST
http://localhost:11434/api/generate
-H "Content-Type: application/json"
-d '{
"stream": false,
"model": "llama3.2-vision:11b",
"prompt": "What is the solution to the equation hidden inside the image",
"images": ["iVBORw0KGgoAAAANSUhEUgAAAMgAAABkCAYAAADDhn8LAAAABmJLR0QA/wD/AP+gvaeTAAAfSklEQVR4nO2de1BU1x3Hv7vCgmiVpaCogFgUxaBp0cQYo43Y4LOVjtQaMxGTIAoCOhqV0MSUpokZO1iRCBjFxAapJq2PViXjGB8kYqT4qFVQlPiKgDzEiCAK7PaP7jk9e/c+du/eu7ug3xlm9sVylt3ffs/nfM9DYzQajXiiJ3oiXrkBwJ07dxAVFeXstsjS3/72NwQHBwMAzp8/j/nz5zu7SWbKyMjAz3/+cwCA0WjEhAkT8ODBA4vHxcfHIz4+nj4uOjoat27dcnh7O6siIyOxdu1aFBcXIyUlhfcxOp0Ox48fh0ajwalTp7Bw4ULJ59UCgF6vV77FDtKAAQPo5UGDBjm1LXwKCwujlzUaDdatW8f7uBdeeAEA0N7eDo1GgzVr1jisjV1Bw4YNAwBcvHhR8DFZWVnQaDQwGo1YunSpVc+rhemNe/bZZ5Vqq8Pk6+sLd3d3et3Ly8up7eGTj48PAODRo0cAgNGjR0On05k9xtfXF2FhYWhtbUV2djYAYPjw4bRonkhaw4cPBwCUlZXx3q/T6RAREQEAOH36NK+L80lLLixatEiZljpQ3A+QRqNxWluE5ObmBgBYvXo1jEYjNBoNMjMzzR7zwgsvQKPRoKSkBPn5+bh79y4A4O233zb7AngiYQ0dOhQQcRA57gEA2vb2doDTFegs+vWvf23Vbc5SZGQkvXzo0CGcPn0a4HGR8ePHAwCKiopgMBiwYcMGwOQsr776qsPb3dnk7+8PvV6PxsZG1NTUWNwv1z0AQFtRUQEAcHd3R9++fRVstrry8PCg/c729naQwbiZM2c6uWX/1/Tp0wEAbW1tAIDk5GQLF/Hw8MCYMWNgNBpx/PhxAMC+fftw+/ZtAMCCBQvQr18/p72GziDSvSovL+e9X657AID222+/pVeSk5PtbKrj9Mwzz6Bbt24AgIaGBrS0tAAuBurEle/cuQOYOIR1ke7du+OZZ56Bp6cnysvLUVdXBwAwGAyURdzd3W1+Ux83kS9KvgKxxz0AQMs+6fPPP293Yx2lCRMm0MsXLlzA1atXARcDdQLoFy5coLexLrJu3Tr6OoqKisx+t7CwEN9//z0AYNKkSZ3qvXG0xBzEHvcAt0B69epFv5VdWRqNxgzQ9+3bhz179tD7XEUE0Pft20dv47oIyx+sDAYDNm/eTK+vWLHCYvTrif4nAujcArHXPQBAW1NTg8bGRnrDvHnz7G6w2ho6dCj69OlDrxcVFdECAYDo6Ggntez/YgGd++FnXcTPzw+1tbW4fPmyxXMUFhbixo0bAIDAwEC8/PLLDmh551K/fv2g1+tx9+5dym1E9roHyDAvOzTmSqNAQmK7VwSAYUqg4SIFwgV0VqyLAEBxcTH4ZvwYDAbk5eXR60+A3VKE87j5hxLuAVIgrDX5+/vb016HiO1eEQAG4FKgzgV0rtgBEdKH5lNhYSGuX78OAPD09MSSJUsUb2tnlhCgK+Ee4CsQrVbr0kBIUmciFoBdCdT5AJ1Vr1696OUhQ4age/fuvI8zGAzYunUrYHLIX/ziFy79/jhafICulHuAr0AAIC4uTvYTqq3x48ebgTgLwK4E6nyAzorAOSTmaIFxEfK63nzzzSfAbhIfoCvlHiAFwgV1YluuKPaDBQ4AuwqoiwE6EXkdpPtEchE+sS7S1taGoKCgJ8AuAOhKugfYuVgsqOt0OpdM1T08PMwmVfIBsCuAuhigg3kdRqPRIhcREnERMjfrCbDzA7qS7gG2QIhFkQ+YK6bqJHUmHzw+AHYFUJcCdDY9r6qqskjX+cS6SEtLyxNg5wF0pd0DfAVCpmW7IgiS4V0SZvIBsCuAuhSgc9NzW13Ey8sLjx49euyBnQvoSrsHxBzE1VJ1Nj3Xav/XbD4AdgVQFwN09nWQAuGbo8Un1kXIt+PjDOwsoKvhHmALhIC6p6cnvdOVUnWSnv/www/0Nj4AdjaoSwE6eR3c9NxWF+nduzfq6+sfW2DnAroa7gG2QMCAOunHu1KqTrolzc3NgAgAw8mgLgXobPeKTc/luAj5/ccR2FlAV8s9wC0Q0s2qrq4GXCxVJ90SwhZCAAwng7oUoJPX8fXXX1vcZ6uL+Pn5oays7LEEdhbQ1XIPCBUIWfLpKqk6u2a7Z8+egAgAw8mgLgbo7Os4deqUxf1yXMTb2xsPHjx47ICdAPrly5dVcw8IFUhISAgdzXKFVJ2k5ydPnpRMqOFkUBdrH/s6WltbeX/fVhfp378/XYn4OAE7AfRXXnlFNfcAt0AIqHt7e6OyshJwkVSdpM5kxR1EEmo4EdSlAF1o7QcrOS4ybNgwXLt27bEBdhbQw8PDAZXcA9wCAQPqpI/s7FSdTZ3JHlhigE7kDFAXA3T2dRQXF4s+j60uEhAQgGPHjgGPCbATztNqtaq6B/gKhHSz3NzcYDAYACen6s8++yw8PT1RVlaGkJAQQALQiZwB6mKATtLzsrIyMyfkkxwXiYyMxOHDhx8LYCe9GjIjWi33gFiBhIWF0S1UnAl/pFvy9ddfSybUrJwB6mLtI8O7fKNXfLLVRQIDA3HmzBm0tLR0eWBn18+o6R6QKpDdu3cDTkzVuamzNYBO5AxQF2ofX3ouJTkuMnv2bHz66adAFwd2lovVdA/wFQgL6gcPHqS3OyNVZ1PnwMBAers1HzJHg7oYoAul51KS4yL19fVdGtj79esHb29vwAHuAb4CAQPqoaGhuHfvHuCkVJ1NnadOnQpYCehEjgR1MUAXSs+lJMdFXnvtNWRkZABdFNhHjBhBL6vtHhAqELabRUZcnJGqs6kz6XdaA+hEjgR1MUAXS8+lJMdFfHx88NVXX3VJYGePNnDEhnqSBUK2yHR0qs5NnW0BdCJHgrpQ+6TScynJcZG4uDisX7++ywG7TqejX9SVlZWquwesKZC6ujqnpOrc1NkWQCdyJKgLtc+a9FxKclwkIiKCFkxXAfasrCx6eeXKlQ75m7wFwoK6v78/ZRJHpups6mzNGm8+OQrUxdpnTXouJbkusmPHji4D7OyM3fb2drqWX23xFggYUA8LC0Nubi5tpCNSdW7qPG3aNMBGQCdyBKgLAbot6bmU5LjIpEmTsHbtWqALADuZsQtAVldVrgQLhO1mlZaW0g9adnY28vLy8M0336C0tBSlpaV44403FG0Um57X1dXJAnQiR4C6EKDbkp5LSa6LnDp1qtMDO+seMJ1F6ShZVSB/+tOfaPUOHDgQTz/9tNnKQ6XFpuewYo23mBwB6kLtszU9l5IcF5kyZQoyMjI6NbCz7gGRc0DUkFUFQtaAE/3www+oqqpSpUF8qbMcQCdyBKjztU9Oei4luS7S0NDQaYGdyx5wlQJhQf3GjRt0CgMA5OfnY8eOHao0iJs6ywV0IrVBXah9ctNzKcl1ke3btzsd2EeMGIH4+HisW7cOW7duxccff4z09HRMnjxZ8CxGdrWgm5sb7y7uakqwQMCA+tmzZ/HRRx85JFXnps72ADqRmqAuBOhy03MpyXURg8HgNGAfP348duzYgU8++QTx8fGYMGECRo4ciYiICEyfPh3vv/8+CgoKLDiRdY/vvvsOEDnFVi2JFgjbzYJpm36onKpzU2d7AJ1ITVAXAnS56fmaNWuwd+9e0R/y/7fVRUpKSpwC7EuWLMHgwYMBAPfu3UNxcTEOHjyIsrIy+uUxaNAgZGdnm23qzboH+ew5snsFWwuETdXV+LDxpc72ADqRmqDO1z570nM/Pz8MGDBA8ofIFhfp1q2b04D9woULWLFiBV566SWkpKQgLS0N8+bNw2uvvUb3QPDz88OcOXMAnl0ShwwZArh6gbCp+pgxYxRvDJs6BwcHY+nSpRSAAwICsHz5crPJatZKTVDnA3Ql0vOOjg4cOnRI8OfIkSOADBepra11OLCvW7cO8+fPx5EjR9DR0WF23/nz5+mx1wAwatQogGeXRKFj1tSWm9idBNT1ej38/f1RU1ODixcvYuTIkaoEhmR490c/+hE+++wzsw90aGgoQkND8fLLL6OwsBBr1qyhXScp7dmzB2+//TZg4hAW3O2REKArkZ4/fPgQqampoo/ZtGkTRo0aRV2Eb24ScZH09HQkJibi1q1bOHfuHKqrqxEUFIRly5bhwIEDNrXNYDDYlEVIhaTsxundu3e3cA9vb2/BY9bUlmiBwNT4sWPH0hWGubm5yM7OVnwBFbtzO/nnkNEag8GAY8eOYdy4cdDpdJg6dSp69+6NZcuW0aE/KZHnUrJA+ABdyfRcSsnJyTh+/Dh1kYSEBN7HFRYW4vXXX8fAgQOxZcsWs/tiYmIQExNj09998OCBxTEU9ig0NJRevnLlioV7jB07FnACoEOqiwWeblZJSQldq66kSHpOVFNTQ8G3rq4OK1asQExMDJ2D8/zzz2P27NlWP78aoM4H6NxZAGpKzoiWq8nX1xeJiYmAqZ179+612OdK7Bx0tSXpINwCgenD279/f0Ubwv1Geuedd5CTkwMwAFxVVYVVq1Zh+/bt6NatG15//XX8/e9/x8OHDyWf/+rVqwgPD1cU1PkAnTsLQG3Z4iKJiYno27cvtm3bhm+//RZ6vR7vvvsuPDw88NFHH1k9EMLlCLny8/NDbm4uPbE4JycHiYmJFjuViJ2DrqZ8fHzkFcju3buxePFixRqi0WgwceJEev3MmTM4c+YMLwBfuXIFRUVFmDhxIry9vfHcc8/RLW/EtGfPHoSHhysK6tz2KZmeu7m5ITk5GU899RTdPbG6uhqnT5/GwYMHaSZFXMQaFsnOzkZ6ejoiIyORnZ2Njo4O9O/fH0lJSfjVr36FgoICOgijtvr06YNNmzbRpdTbtm3D9u3bqZuwqwWdAeh6vR7Z2dnSXSzu1HcA+Mtf/qJoY4YOHQq9Xk+vHz16VDRBP3r0KL3MnngrJqUTdb72KZme63Q6xMbGYvTo0Rg8eDBGjBiBqKgopKamYt++fZg7dy4tdrnpOgCnJOy9evXCxo0baXFs3boVWVlZvHvsip2Drpb0ej1ycnIwePBgaQcBD6h3dHSgtbXVqgmLAQEBCA8PR1BQED34paqqCv/+979pPsGeew7T0N+rr74KCCTo//nPf+hlEkARDR8+HCEhIfDx8UFjYyOuXr2K8+fPw2g0KgrqfAm/kun5o0ePUFNTg9u3b6OtrQ3+/v4YNGgQNBoNvLy8sGzZMgQFBeHDDz+02kXCwsJw9OhRxMbGYvHixaivr6f9/iVLlmDhwoVm7Cekjo4Os3PebdXSpUspCxYUFCA7O1twh3ahc9DVko+PD3JychASEoLKykrrCqS8vJwWCBl/v3btmuACqsDAQERFRWHSpElmIxRcnTt3DhkZGRYuUFVVJZqgV1dXw2AwmAWWo0aNQmpqKi+E37p1C2vXrkVLSwt69OihCKjztc+etedEJ0+exOeff46ioiKLDIV0h6KiogDTCNS5c+dw4MABq1jk3XffxU9+8hPA1MXZuHGj2f06nQ7vv/++ZBvtGcXS6XSYPHkyYHofSQYitEO7IwGddKtCQkJw7do1JCYmSnexwFk8RcR2e4KCgswen5CQgISEBIvi4PZvR44cia1btyIsLMzsG7ehoYECMN83R1tbG5qamgAAPXv2xLRp05CTk0M/+EajEXfv3qWjbQMGDEBmZiY9fEcJUOcCur1rz4k2b96MgwcP8gaMVVVVSEtLw86dO+ltixYtgkajsXpEy9nq27cvPDw8AFNPob29XfR8D0cBOtutunbtGhYuXIiGhgbrHIR8SNkCuX//Pr1M0k+url+/jgMHDqCkpASXLl3Co0eP4O3tjfDwcCxatAjDhg2jeQopkLa2NhgMBgrA//znP3mfu7W1Fb179wZMI15arRYGgwFbtmzB9u3b0dzcjO7du2PWrFlISkqCm5ub2Rwme8UFdCXSc2u1YcMGvPTSS/Dx8UH//v0xZMgQVFRUSLpIeno6vLy8oNVqsXr1arMRLQCYPHkyoqOjUVtbi/fee08wY7JnFIstXFIIYud7OALQhYoD1gzzQiBRZ+Xn52d2vaysDPv27eMNyu7evYtvvvkG//rXv5Cbm2sxdcRgMFg1xZ19k8hU6ezsbLNp+Q8ePEB+fj6am5vxu9/9zmxdiz0cwtc+JdJza/Xw4UOcOHGCBpWhoaGoqKiQZBF2GJdvROvs2bP46U9/iuDgYISGhmLbtm2qvg6DwSDqHo4AdLHigDVBIUxdKNLw5cuXIzk52YwbunXrhtTUVCQnJyM5ORl6vd6iiLh6+PAhXevOSqfTWTXFnTtAUFtbi88++4z3sbt376bHORDZM5I1f/58wPQGp6en449//CPGjRsHmALMkSNHyn5ua8V+YIiTws4Rrba2NtWnxFdXVyM1NRWpqanYtWuXqHuoDehSxQFrCyQgIIAGgxMnTkRsbCydFkIUExOD2NhY+sPlEj716NHD4jaNRoOnnnoKEJnirtFo6ElTRF9++aWo9e/fv9/suj2gPnDgQMA0q3n69OmYMmUK7SpOmjQJAQEBsp/bWrGvn/3WlbtehLRf7SnxTU1NdMLllStXRE+HUhPQrSkOWFsgaunFF1+kl1m3IJmI0DfHj3/8Y4tZqGfOnBH9W9xhSXtAnXzo7t+/j4qKCgr/DQ0NZtfVFLuJwa1bt8zus8dFAKg6JT4wMJD2NAoKCkTP91AL0K0tDlhbIMXFxZgxYwZgYojRo0fTH3aocMyYMfR2NszjE5s6w8Q5RFKATsCN1bVr10T/Hslc2L8vV4RlVq9ejVdeeYVOdUlOTsbcuXPpkWhqKSoqiuY/ra2tOHv2rNn99rqImlPi+/XrR3sZwcHBgMgeu2oAui3FAVschC9RBydVt2UH+Kefftps9RjflGsh4H3uuecsbqutrRX9e83NzRbT4+VwCBfQlUzP586di1GjRokWb1RUFN555x16/YsvvuCdi2avizgqYRdyDzUA3dbigK1dLL48pKOjw+a16hqNxmytQ2FhIQoKCszeaKEhRi8vLwrxRAaDwaoJi0oUCHcAQcn0PCIiAps2bcL+/fvx3nvv4Y033sCsWbPwm9/8BikpKfjrX/+KDz74gDpCRUUFNm/ezPtc9rqII4AdIu6hNKDLKQ7YWiB8ExchY616XFwc7SLU1dXhww8/REtLi1kA1tHRYbHThVarRVpamtmoDXgCSCFxHycH1LkJuhLpOVd9+vTB1KlTkZCQgLfeegurVq3CvHnz6LJTADhx4gSSkpJEF43Z6yJqAPuFCxfo50jsfA8lAV1ucUAJBwFnrboU1M2dOxcLFy4EmH9Qc3MzAOCTTz6h38IeHh7Iy8vDjBkzEBERgalTp+Ljjz+mb2B9fT39Fre2j8wtODmgziboSqXnRJs3b0Zubi5OnjxJXZlVQ0MDDh8+jJSUFKSkpEjOmbLXRaACsLe1tdEPv9j5HkoBuj3FAWuDQiK+RB3MWnWdToe4uDjBlXRz5szBsmXL6PXS0lJcunSJXm9qajLrfw8fPhy///3vLZ6nrq4OS5cuxcaNG+Ht7Q2tVgudTifpJKQgyDwuOaDOJuhKp+eXLl0y+3+4u7vD29sb7u7uuHfvntnsBWslZ9XhlClT6LA4AfakpCS8+eabmDNnjl1T4sVyD1ZKALq9xQFbHUQI1MG4i9AExtmzZ2P58uUA03//8ssvzR7DAvDWrVtRXV1tdn9tbS3y8/Px29/+FpcuXUJ9fT29jyy6EVKPHj1o7sL+ni0cwgV0tdPztrY21NXVoaqqSlZxQCEXUQrYxVJzVkoAuhLFATk5iFA3S2wH+FmzZmHFihXQaDS4d+8eDAYD75ptFoCzs7Pxy1/+Ei+++CJmzpyJyMhITJs2DevXr6fdD7KZGJjwTkjs/d99952szeTY9jly7bm9spdFlAJ2a93DXkBXqjggp0CEQJ1dq86eqx4dHY1Vq1ZBo9GgqakJW7ZsgYeHB++abb4p5Pfv38etW7d4++Ts3KKf/exnou1mJ1SWlZXJWqPOts+Ra8/tlRIuYi+wW+sesBPQlSwOKOkgYMI+AnMzZ85EWloatFot7t+/j8WLF9NwiG/UR2yKO5/I2hQAmDJlisUm26zYb8QjR47I2kyOBXRHrz23V/a6COwEdmvdA3YAutLFATkFIgTqME0KhGlJ5cyZM+kMWlIc5eXlomu2pRJ0rqqqquhUbX9/f8H+8YwZMyj0lZWV4eLFi7I2kyPt279/v+I7t6stJVxEbsJui3tAJqCrURyQUyBioM6m6qQ4Wltb8dZbb+HmzZuIiIhAnz59UF9fj9u3b6NXr170h0zdho0fuqysLBoqLlmyBPPnz6dvvk6nw+zZs5GWlgaY3vwNGzbAaDTavEadBfTbt2+rsnO72lLCReQAuy3uIQfQ1SoOyJ2sKNTNYlN10t3x9PREVlYWDh8+jE2bNgGm1XdfffUVDh8+TH/S09MBGbu4X7p0CR988AGMRiO0Wi2SkpJw5MgRHDhwAMeOHcPKlSvpN92GDRtQWlpKf9cWUGcBXa2d29WWEi5iK7Db6h62ArqaxQG5BSIE6rBim0kpydnF/R//+AdSUlLorFY3Nzf06dOHBoO1tbVYuXIl8vPzzX7PFlBnAV2N9NxRUsJFbAF2W9wDNgK62sUBW4NCIjFQz8zMpP/QnTt30oVKPXv2REpKCtra2rB+/XoLp0hNTYVWq5U9tHfixAlER0djxIgRGDp0KHr16oWmpiZcuXIFZ8+e5d0N0pbN5AigX758GePGjVMsPXe0bNlHi+zpGxcXZ7HeJiMjA2PHjqXAzvfFaKt7wAZAd0RxQK6DiIE6uwN8WFgYdu3ahV27dtFNFoqLi7Fz5056O/khXTJrAZ1PRqMR586dwxdffIG8vDx8/vnnOH36tOBWqbaAOgH0qqoqh609V0tKuIg1wG6re8BKQHdUcUBugYiBOgRSdbHU2d5j1uTKWlBn20deb2cZveKTEiwCCWCX4x7WALojiwP2rCgU62ZxU3Wp1FmJY9bkyhpQZ9vXWdJzKSnhImLALsc9pADd0cUBewpEDNS5qbpU6qzEMWtyZQ2ok/Y1Nzd3mvRcSkq5CB+wy3EPSAC6M4oDajkIOKm6VOpsa4KupKxJ1En7yLT8zjh6xSclXAQ8Cbsc94AIoDurOGBPgYiBOjipulTqbGuCrqSsAXXSPjIbuDPzByulXIQF9hUrVshyDwgAujOLA/YUiBSos6m6WOrsLEAnkgJ1tn3e3t6dLj2XklIuQoA9MDBQlnvwAbqziwP2bvsj1s1iU3WIpM7OBHQiMVAn7SMZQGdLz6WklIu0tbXhz3/+M71+/vx5m9yDC+iuUBywt0DEQB2A2fY3Qu7gTEAnEgN10j4y6NBVuleslHIRdlcbdlGaNWIB3VWKA2o6CEynBhFx14MTORPQicRAnbTP3d2906bnUlLCRdiRK4PBgIkTJ9o0JZ58Ed28edNligP2FogUqLMbU8fGxvI+xpmATiQG6qR9MJ3dYc32Qp1R9roIO3JFtiKyZUo8AfT58+e7THHA3gKRAnX2kBW+terOBnQiIVBn24cu2r0issdFuLnHp59+atOUeALoHR0dCA4OdpnigBJ78wp1s9j0HAJr1V0B0In4QJ3doK4rpOdSkusi3NzD1inxo0ePBkynBLhScUCJAhECdTY951urDhcBdCI+UCftg6k72dnTcynJcZEFCxbw5h7WTonX6/VISUkBTPs+u1JxQE0HYdNz7lp1IlcAdCI+UCftQxdKz6Vkq4sEBAQI5h5Sa9jJaBXZzT8zM9OligNKFAgfqHPPC2dTdXbkwxUAnYgP1FlA78r8wcoWF2HDYL7UXGxKPDuUSzKmkpISVV6TPbK7QPhAnbvjOd8O8K4C6ERcUGfb19XScylZ6yIso3E3ASTimxLPFsf333+Pbt26OfQcdFukyAE63G4Wd8023w7wrgToRCyos29+V0vPpWSNi7AjVzB98XHTdfBMiR82bJhZzkEcxhW62XxSpEC4oM63OIqk6sRlXAnQiVhQZwHdFRzO0ZJyEXbk6saNG4LpOjjAnpuba5ZzkOPqHHEOuhwp7iC+vr4YNmyYRepM5umQHeBdCdCJWFAn7evo6OiS6bmUxFyEm3vk5eUBAnO0iPLy8mAwGNCzZ0/cvn2bjlY56hx0uVKkQFhQZ3c8Z1PnO3fu0LXqCxYsoABcWFioRBMUEQvqpH0XL17ssum5lFgXWb9+Pb2ddY/k5GTJOVp6vR5/+MMf6L4D7e3tdI8CR5yDbo8UKRAW1GNiYgCBYVHiNOHh4fS2w4cPK9EERcR3bjoZgXscxbpIREQEdDqdhXs8evRIdI4WC+TXr1/HzZs3MWDAAMydOxf+/v7Q6/VobGx0SUCHkqfckg//0KFDYTQaeQ+yJGvVyVCqKwE6ERfG1T6Q09XFukhmZqbgakE+F+HOyo2Pj8eaNWsAUyGRkUJXdQ8oWSDsiywvL+dNndm16gDQ2Nio1J9XTOyRZuR8jsdZXBYRWi3IdRFfX1/eWbklJSU4dOgQPD09kZSUBDBfrq4oxR0EEqM+7HHP7PEFriL2uOibN286tS2uItZFxFYLFhYW0hGtnTt3YvDgwaisrER8fLxZQr5u3Tq0tLTQ4NCVBmq4UsVBxKZlsH16VwJ0or179/JefpzFughE1pobDAY6otW7d29UVlYiISHBYii/traWPg4u7iD/BdFaLabPQ77dAAAAAElFTkSuQmCC"]
}'
Learning points
- The images field is an array of base64 encoded strings sans the png header
- Set the stream field to false for convenience
- Set the appropriate model and prompt. I'm using llama3.2-vision because it supports images.
Okay, things are starting to take shape...
Let's start to pull it altogether now and let's create captcha-runner.js.
Below is the finished version:
captcha-runner.js
const axios = require("axios");
const { generateCaptcha } = require("../library/captcha");
function queryOllama(
image,
prompt = "Please transcribe the simple mathematical problem expressed in the captcha. Do not include anything else.",
model = "llama3.2-vision:11b",
context = "You are a helpful bot trying to test the effectiveness of our captcha images. Please provide a transcription of the text hidden in the image. No summary or preamble required.",
) {
return axios
.post("http://host.docker.internal:11434/api/generate", {
model: model,
stream: false,
prompt: context + prompt,
images: [image],
})
.then((response) => response.data.response)
.catch((error) => {
console.error("API call failed:", error.message);
process.exit(1);
});
}
let weights = {
dots: 100
};
const generateCaptchaAndTest = async (weights, counter) => {
return new Promise((resolve, reject) => {
const captcha = generateCaptcha(weights);
await createPngFromBase64(captcha.image, `captcha-${counter}.png`);
queryOllama(captcha.image.replace("data:image/png;base64,", ""))
.then((response) => {
console.log(response);
console.log(captcha.equation);
resolve({
response,
equation: captcha.equation,
weights,
unsolved: response !== captcha.equation,
});
})
.catch((e) => {
reject(e);
});
});
};
async function keepTrying() {
let trying = true;
let counter = 0;
while (trying) {
try {
counter += 1;
const result = await generateCaptchaAndTest(weights, counter);
console.log(`Weights: ${JSON.stringify(weights)}`);
console.log(`Unsolved: ${result.unsolved}`);
if (result.unsolved) {
console.log("Success!!!");
trying = false;
break;
} else {
weights.dots += 50;
console.log(`Incrementing weights: ${JSON.stringify(weights)}`);
// Add a delay to avoid overwhelming the API
await new Promise((resolve) => setTimeout(resolve, 1000));
}
} catch (error) {
console.error("An error occurred:", error);
weights.dots += 50;
await new Promise((resolve) => setTimeout(resolve, 1000));
}
}
}
keepTrying();
Learning points
- host.docker.internal maps to the localhost on your host machine (I'm running the code inside Docker).
- Ollama doesn't need the "data:image/png;base64," blurb inside the base64 encoded string (just in case you have it, like I do)
- Rather than testing the solution itself, I wanted to test its ability to transcribe the distorted text.
- "Prompt Engineering" is more of an art than a science. Play around with your wording until you start getting acceptable results.
After making changes to the captcha.js code so that it would accept weights when called, we were able to generate and test and then increment the weights until the model failed.
Example output from the command line.
node captcha-runner.js
Image saved to captcha-1.png
Model Transcription: 400 - 3 = ?
Captcha: 400 - 3 = ?
Weights: {"dots":100}
Solved: false
Incrementing weights: {"dots":150}
Image saved to captcha-2.png
Model Transcription: 50 + 3 = ?
Captcha: 50 + 3 = ?
Weights: {"dots":150}
Solved: false
Incrementing weights: {"dots":200}
Image saved to captcha-3.png
Model Transcription: 30 + 10 = ?
Captcha: 300 + 10 = ?
Weights: {"dots":200}
Solved: true
Successfully solved!
Wahoo!!! We've got something that the model failed to correctly solve!
Let's take a peek!
To ensure it was bullet-proof, I repeated the process a number of times. Sometimes the LLM performed better and so the starting point for the number of dots increased each time. Eventually, however, it at settled around 800 dots.
