2024-09-08T10:59:52 *** teepee_ is now known as teepee 2024-09-08T17:00:05 hendersj: can check now when you're up. enabled debug mode and tried it myself two times but (of course) couldn't reproduce 2024-09-08T17:02:48 whilst at it I find some stacktraces which unsure if related but surely not quite right 2024-09-08T17:05:19 Yeah, I was unable to reproduce it as well just a few minutes ago. I was looking to see if I could find a Firefox plugin to trace oidc traffic more clearly. 2024-09-08T17:06:39 There might be some irony in having debug mode enabled causing it to just not happen. I was diagnosing a memory leak in a different IdP (the one I use here at home; a commercial solution that I have a license for), and when I turned on the option to track memory utilization, it stopped leaking memory. Love it. 2024-09-08T17:08:45 classic 2024-09-08T17:09:14 one of the peculiar exceptions I find 2024-09-08T17:09:22 sqlalchemy.exc.InternalError: (MySQLdb._exceptions.InternalError) (3, 'Error writing file \\'/tmp/#sql/fd=309\\' (Errcode: 28 "No space left on device")') but there is gigabytes of free space on both the ipsilon and the MariaDB servers 2024-09-08T17:10:27 Huh 2024-09-08T17:11:00 Is /tmp just tmpfs or is it actual disk space? 2024-09-08T17:11:14 it's real disk 2024-09-08T17:11:41 Obvious question, maybe, but no indication of a hw issue? 2024-09-08T17:11:41 but it's on the root file system so it only has a few gigabytes. maybe it writes something really big but immediately deletes it again when it filled up 2024-09-08T17:11:50 That could be 2024-09-08T17:12:05 these hypervisors run many other VMs 2024-09-08T17:12:14 Makes sense 2024-09-08T17:12:50 I mean, if it was hitting an i/o error, there would be other indications, and 'disk full' is probably not what it would get 2024-09-08T17:14:42 I'm just comparing a successful auth with the failure from yesterday, and the only real difference seems to be the response to "id.o.o/Continue?ipsilon_transaction_id=xxxxxxx" giving a response header that points to the correct redirect URI (in the working one) as opposed to the wrong redirect URI in the broken one. 2024-09-08T17:16:31 Before that, I see a POST to "/login/ldap" - not sure what that is for. 2024-09-08T17:17:25 Whatever is being posted is probably being sent as JSON, and the developer console doesn't capture the payload, only the headers and parameters. 2024-09-08T17:18:11 But the response headers on that POST include a 'location' header that is the next call, which is to "/Continue" with the ipsilon_transaction_id populated. 2024-09-08T17:18:23 * "ipsilon_transaction_id" 2024-09-08T17:19:16 So it seems logical that it's requesting that value via an LDAP lookup...but that also seems odd, because that would mean it's using an LDAP directory as a write-heavy temporary storage space, which is not what LDAP directories are optimized for. 2024-09-08T17:20:24 hm but ipsilon cannot write to ldap, or more like I'm rather sure the service user can only read 2024-09-08T17:20:38 Yeah, that also makes sense. 2024-09-08T17:20:55 I mean, I guess a POST can be used like a GET anyways 2024-09-08T17:21:06 I have an idea about how to grab the payloads, let me try something. 2024-09-08T17:21:29 can != should :P 2024-09-08T17:26:12 True enough. Anyways, my idea worked (there's a useful took called httptoolkit that is essentially a mitm tool for diagnosing https issues). The POST is sending the transaction ID, login_name, login_password, and login_otp (if any) through the "/login/ldap" endpoint - it's the login form page submission. 2024-09-08T17:26:41 ah, interesting. then just poor naming of the path 2024-09-08T17:26:48 Yeah, seems to be. 2024-09-08T17:26:59 Maybe the 'ldap' part is just indicating the method rather than the protocol 2024-09-08T17:27:07 true 2024-09-08T17:27:15 If it were gssapi, it'd probably be "/login/gssapi" 2024-09-08T17:27:27 Or even the name of the object in the configuration 2024-09-08T17:28:35 In any event, that mystery is solved. The response body basically contains the value of the response 'location' header, but with some additional text ("This resource can be found at ") 2024-09-08T17:28:41 right just the name of the login plugin 2024-09-08T17:28:50 Yep 2024-09-08T17:29:00 the admin path to configure the ldap backend is /admin/loginstack/login/ldap which matches that 2024-09-08T17:29:11 That makes sense 2024-09-08T17:29:35 So at least we don't have ldap diagnosis to do as well. That's something. 2024-09-08T17:29:48 So it does seem to come back to the transaction ID lookup coming back with the wrong value. 2024-09-08T17:30:13 I was trying to parse the code yesterday afternoon to see if that made sense in the flow, but I didn't get very far with it 2024-09-08T17:42:11 I tried multpiple times with different accounts and different services now (always fully logging out of id.o.o in between) and can not reproduce it 2024-09-08T17:42:16 never did I want to see a 500 error so much 2024-09-08T17:42:37 Same here. I'm just looping through logging in and out and...nothing 2024-09-08T17:42:38 but I can leave debug=True enabled for now 2024-09-08T17:44:28 Yeah, doesn't seem to be hurting performance too much, so hopefully when it happens again we'll have more info to go on. And knowing the user who has the issue and the time they had it will help find the right part of the log. I guess it's just important to make sure the logs don't fill up the free space 2024-09-08T17:45:50 right. will need to keep an eye, there is no monitoring of the ipsilon machine 2024-09-08T17:46:06 In the meantime, httptoolkit is definitely worth checking out for helping diagnose issues like this. When I was at Ping, we used to use a Windows-based tool called Fiddler for doing similar things. httptoolkit is open source (there's a paid "pro" version, but the pro features are also open source, so it can be fully built and run with all features - but I find the free version is sufficient) 2024-09-08T17:46:22 fiddler I vaguely remember from school 2024-09-08T17:46:30 I kinda wish I'd remembered it yesterday. Don't have much need for it normally 2024-09-08T17:46:46 now I only do browser console debugging which works but can sometimes be a bit finicky to find the right thing at the right time ;) 2024-09-08T17:47:06 Yeah, and usually that doesn't include the payload, which can be very helpful 2024-09-08T17:47:22 in firefox it does 2024-09-08T17:47:35 Hmmm, I didn't see the payloads in my traces 2024-09-08T17:48:00 Network -> click on POST request for "ldap" -> Request 2024-09-08T17:48:04 shows the "form data" 2024-09-08T17:49:39 example: https://paste.opensuse.org/pastes/7b62795c3ea7 2024-09-08T17:49:48 Ah, I hadn't seen that before. Good to know. Seems that it returns some stuff, but not the full payload. 2024-09-08T17:50:19 ie, it won't give you the javascript for a GET of jquery.slim.js (one that I checked) 2024-09-08T17:50:23 But yeah, for form data that makes sense 2024-09-08T17:50:28 that is not part of the POST payload 2024-09-08T17:50:38 jquery.slim.js GET is also there 2024-09-08T17:51:05 I was looking at the response data tab, and it's not showing it for me 2024-09-08T17:51:27 But I see some things, like favicon.ico and svg do come through. 2024-09-08T17:51:38 this? https://paste.opensuse.org/pastes/81c0ca1bc76a 2024-09-08T17:52:22 Weird. Mine shows "No response data available for this request" 2024-09-08T17:52:55 try it with disable cache and with the network tab open while you load it 2024-09-08T17:53:21 or alternatively right click on the GET line and "Resend" 2024-09-08T17:53:44 https://paste.opensuse.org/pastes/d2bf9f8902e2 2024-09-08T17:53:49 Will do - that's a good tip to know 2024-09-08T17:55:44 Yeah, the cache setting was it. I don't spend a lot of time in FF's tools, and rarely in Chrome's these days. :) 2024-09-08T17:58:01 I am coming from Firebug 2024-09-08T18:00:05 Is that still maintained? I thought it was discontinued 2024-09-08T18:00:38 yes I mean I used that before firefox developer tools already ^^ 2024-09-08T18:00:46 I think in between was firefox developer edition 2024-09-08T18:01:48 Ah, I misread :) 2024-09-08T18:01:59 Yeah, I remember using firebug years ago 2024-09-08T18:02:27 I spent far more time doing just general network troubleshooting with tools like Network General's Sniffer, LANAlyzer for Windows, and now Wireshark. 2024-09-08T18:02:42 But I don't really do much with Wireshark these days any more either. 2024-09-08T18:03:36 I feed tcpdump into wireshark when I want easier readable output. or to decode SNMP 2024-09-08T18:19:56 Yeah, I've been known to do that as well. Occasionally use tcpdump on my router when diagnosing issues 2024-09-08T19:55:21 *** teepee_ is now known as teepee 2024-09-08T20:19:40 acidsys: Is there a backtrace with this message? 2024-09-08T20:21:26 which? 2024-09-08T20:24:24 The no space error. Doing a little reading, it seems like it could be related to insufficient inodes (which I'd be surprised if that was the issue on a dedicated system for this) or maybe something other than space - like system limits in limits.conf, perhaps? 2024-09-08T20:24:49 But I wondered if there was a backtrace to see if I could work backwards up the stack to the part of the code to see if it makes sense related to the results we see when it fails 2024-09-08T20:25:30 sqlalchemy is used in data.py in the util directory of the code, but it's just a general library used by various things in the rest of the code 2024-09-08T20:29:04 going with the monitoring output I pasted, it does actually run out of space briefly every so often 2024-09-08T20:29:35 but yes can paste the whole thing, wonder what it is querying from MySQL that causes mysqld to write such a big temporary? file 2024-09-08T20:30:37 That certainly is strange. I somehow didn't see the paste with the monitoring output, just the error message itself. 2024-09-08T20:31:24 hendersj: https://paste.opensuse.org/pastes/edbb50b14473 2024-09-08T20:32:46 oh, I guess I eventually did not send it. well / on the database server does fill up briefly and then immediately frees up again relatively often 2024-09-08T20:32:53 so I would assume that correlates to the error 2024-09-08T20:33:52 Seems a reasonable assumption. 2024-09-08T20:34:12 and it does correlate to my SELECT queries yesterday which too several minutes to return 2024-09-08T20:34:30 let me try running the exact query as in the error 2024-09-08T20:34:40 I was just thinking that myself 2024-09-08T20:36:48 prefixing the query with EXPLAIN might also help 2024-09-08T20:36:59 Good point 2024-09-08T20:37:20 Unfortunately, the backtrace doesn't seem to point to anything specific - the second one looks to be part of an auto cleanup routine. The first one doesn't come back to anything in the ipsilon code 2024-09-08T20:37:58 (But actually, that makes sense, because it's the mysql handler for sqlalchemy, I think) 2024-09-08T20:39:06 running now together with `i=0; while :; do df / > /dev/shm/df.$i; i=$((i + 1 )); sleep 5; done` .. waiting til it returns 2024-09-08T20:39:16 kk 2024-09-08T20:42:34 returned after (6 min 43.805 sec), but disk did not fill significantly in that time (only a few bytes) 2024-09-08T20:42:59 can try with EXPLAIN, but I wonder if there are some other options sqlalchemy passes 2024-09-08T20:43:58 https://paste.opensuse.org/pastes/92a154878459 2024-09-08T20:44:28 Yeah, I was trying to dupe while you were running the query, but no luck 2024-09-08T20:44:34 same 2024-09-08T20:45:24 that EXPLAIN output is crying for CREATE INDEX ;-) 2024-09-08T20:46:09 that "EXPLAIN"s what it's taking so long 2024-09-08T20:47:41 https://paste.opensuse.org/pastes/6540c9aa0384 ;-)) 2024-09-08T20:48:24 3.5 million rows seems like a lot to me. But I noticed that your query (acidsys) uses "= " rather than "<= " (which the DELETE query uses) 2024-09-08T20:49:30 great cboltz, I see you have a new occupation ;) 2024-09-08T20:49:36 LOL 2024-09-08T20:50:00 hendersj: oh, I didn't realize that was part of the mysql query 2024-09-08T20:50:48 I wonder if this is similar to the issues we had a few months ago with jobs in sidekiq - there's a backlog that it's trying to get through, and it can't because it's running out of space, so the backlog keeps growing 2024-09-08T20:51:34 So when the auto cleanup job tries to kick off, we see the issue because the DELETE query is filling the disk long enough to cause problems for some users, and when it finally crashes, space frees up, and users stop having issues. 2024-09-08T20:52:06 in case you can't get enough, this is the other exception which is occasionally printed (not sure if in any way related) https://paste.opensuse.org/pastes/954e6b013cf1 2024-09-08T20:52:33 the cleanup interval is at 30 2024-09-08T20:52:50 if I could run the df thing while it runs that might be interesting 2024-09-08T20:53:42 That could be, yeah. 2024-09-08T20:53:54 And that casting error seems weird, but probably unrelated. 2024-09-08T20:55:37 but I just found that is seconds not minutes. so it does seem to run often but not always cause this 2024-09-08T20:56:20 Boom 2024-09-08T20:56:30 hard to say if it's really unrelated without looking at the code around - my _guess_ (without looking) is that normally tokens[iden]['expires_at'] is an integer, but for some reason (some error?) it is a string, maybe '' 2024-09-08T20:56:31 It just happened, but I noticed something new, I think 2024-09-08T20:57:18 Yeah, that's my thought 2024-09-08T20:58:28 there is lots of hendersj in these minutes 2024-09-08T20:59:05 also likely unrelated, but yesterday we wondered about the "None None" - this is populated with the usernames 2024-09-08T20:59:13 So what I noticed is that on the IdP login page, there's a banner that shows which client was called. (That's where the incorrect value of 'discourse.opensuse.org' from the config shows up). When it failed, it actually said it was asked to go to src.opensuse.org rather than discourse.opensuse.org. So it's happening before we thought it was. 2024-09-08T20:59:26 "None None" is probably just from having additional debug turned on 2024-09-08T21:00:06 Of course, now it's not failing for me (again), so I can't see if what I thought I saw was what I saw. 2024-09-08T21:01:26 But if I saw what I think I saw, then we should be able to tell when it's going to fail without actually going through the authentication 2024-09-08T21:02:35 https://paste.opensuse.org/pastes/490e1ed410d9 - at the top, you can see 'discourse.opensuse.org'. Pretty sure I saw 'src.opensuse.org' in there before it failed. 2024-09-08T21:02:36 I don't find any errors in your messages.. only some info that you log in to the app discourse.opensuse.org 2024-09-08T21:02:57 when searching for src.opensuse.org I only find some error from some hours ago, https://paste.opensuse.org/pastes/7a7aba6e499e 2024-09-08T21:03:13 that would be interesting 2024-09-08T21:03:59 Yeah, that means that it's associating with the wrong oidc client as well as the redirect URI, which could just be a lookup failure for the transaction. 2024-09-08T21:04:00 your GET request is logged with the right parameters 2024-09-08T21:04:22 https://paste.opensuse.org/pastes/1c37586d9860 2024-09-08T21:04:40 yes 2024-09-08T21:06:30 it does not seem to show what it returns after the get unfortunately 2024-09-08T21:07:02 but it would be interesting to lookup what it associates with the transaction id which is recorded 2024-09-08T21:07:15 Yeah 2024-09-08T21:08:03 but I'm not sure where, the transactions database only has value and uuid 2024-09-08T21:08:06 I was just looking back over my previous failure capture, but I hadn't captured any message bodies at that point (cache wasn't disabled). 2024-09-08T21:08:29 That's probably used for a join lookup somewhere. Maybe? 2024-09-08T21:08:46 there are no other tables in that database 2024-09-08T21:08:55 Hmm 2024-09-08T21:09:21 https://paste.opensuse.org/pastes/96bc0e2f2627 2024-09-08T21:10:07 What does one of the values look like? It's a text field, so it might just be URL-encoded parameters 2024-09-08T21:10:29 Which actually makes sense. I think it's a form of something called POST Preservation 2024-09-08T21:10:41 value is the transaction id 2024-09-08T21:11:01 Hmmm 2024-09-08T21:11:25 or at least I think 2024-09-08T21:11:27 https://paste.opensuse.org/pastes/29d66988b449 2024-09-08T21:11:38 I replaced with matching amount of x's if that is any help 2024-09-08T21:12:34 that is when name is "cookie". the value when name is "expiration_time" is a unix timestamp 2024-09-08T21:13:04 when it is "origintime" it is a somewhat ISO formatted human timestamp 2024-09-08T21:13:41 Can you query the last, say, 5 values? 2024-09-08T21:13:50 s/values/rows 2024-09-08T21:14:10 how many different names do exist in that table? 2024-09-08T21:14:11 ooh I think it matches by uuid 2024-09-08T21:14:16 Come have a drink in the BAR 2024-09-08T21:14:18 let me try to paste this 2024-09-08T21:14:20 (BTW, I was able to confirm, the banner did say 'src.opensuse.org' when it failed. 2024-09-08T21:14:30 * cboltz wonders if that table should actually be 5 or 10 separate tables 2024-09-08T21:14:43 It wouldn't surprise me, cboltz 2024-09-08T21:16:23 But yeah, it looks like the way the primary key is defined is by uuid+name, so there are probably multiple rows per uuid in the table. 2024-09-08T21:17:19 some have it like this: https://paste.opensuse.org/pastes/1f74ad287722. but some of them also have more information: https://paste.opensuse.org/pastes/bf9b9bf5cace 2024-09-08T21:17:57 cboltz: can I find that out in a way other than doing my long chain of != 2024-09-08T21:18:25 (if I just SELECT name FROM transaction), well, I wait again minutes 2024-09-08T21:18:51 I'd think maybe grab that first row's uuid and just search for uuid = 2024-09-08T21:19:07 Oh, which you already did. 🤦 2024-09-08T21:19:12 :) 2024-09-08T21:19:33 I continued with the second one because it seems not all uuids have all the possible "name"s populated 2024-09-08T21:19:53 So the "cookie" value is the transaction ID, then? 2024-09-08T21:20:13 I'm not fully sure, because when I searched for your ID as "value", nothing came up 2024-09-08T21:20:24 but I'm not sure when and how long it's supposed to enter it 2024-09-08T21:20:41 select name, count(name) from transactions group by name; should give a nice overview 2024-09-08T21:20:59 vice versa, when I search for one of the cookie values in error_log, I don't find anything, but might have been from before I enabled debug 2024-09-08T21:21:21 cboltz: thanks! coming up 2024-09-08T21:23:05 That example with the login_return value - the uuid column and the uuid in the value....are they the same value? 2024-09-08T21:23:33 If they are, then that means the uuid column is the transaction ID that's passed in (which makes sense) 2024-09-08T21:25:08 good question, no, I "SELECT * FROM transactions WHERE uuid = ''" and the cookie value I only find in cookie value 2024-09-08T21:25:37 uuid is different from it 2024-09-08T21:27:31 here the "name"s overview https://paste.opensuse.org/pastes/9e0ccd7888e2 2024-09-08T21:30:20 interesting to see that there are always pairs with the same count (cookie and expiration_time slightly differ, but that's probably caused by a failed INSERT or DELETE) 2024-09-08T21:31:14 now I could say something about the database design, but I'm afraid it might violate the code of conduct ;-) 2024-09-08T21:33:09 design as in ipsilon structure or design as in administrative implementation? 2024-09-08T21:34:43 I'm going to agree with cboltz if it's about the ipsilon db design. 2024-09-08T21:34:51 design as in "how to use a database (in a sane way)" 2024-09-08T21:35:04 each pair of names should be a separate table 2024-09-08T21:35:12 ok ;) 2024-09-08T21:35:20 so my wild guess that this table should be 5 tables wasn't too bad ;-) 2024-09-08T21:35:30 yeah I wondered about that .. which is also why I didn't find there was more than cookie values before :D 2024-09-08T21:36:19 It seems that return_url is the redirect URI value. But it does seem to me that the issue is even before the auth takes place, because the wrong client is being selected to service the request. 2024-09-08T21:36:34 but on the bright side, this way it's very easy to export into an excel spreadsheet 2024-09-08T21:36:42 +1 2024-09-08T21:36:52 actually only 4 tables - cookie, expiration_time, origintime and provider have the same count (with a minor diff for cookie) 2024-09-08T21:37:31 And given how transactional this table _should_ be, perhaps it's optimized enough. 2024-09-08T21:37:40 I'm looking forward for a spreadsheet with roughly 42M rows... 2024-09-08T21:37:44 I'm a bit suspect about why there are so many rows of data here 2024-09-08T21:38:27 it's 19G 2024-09-08T21:38:39 which is quite a lot of transactions indeed 2024-09-08T21:38:57 I searched but couldn't find if there's a way to call these cleanup functions manually 2024-09-08T21:40:16 to explain the amount of data - select name, value from transactions where name="origintime" limit 1; <-- shows the oldest entry 2024-09-08T21:41:18 That seems really excessive in size to me. 2024-09-08T21:41:54 It looks like (from the paste at https://paste.opensuse.org/pastes/1f74ad287722) the expiration time is an hour after the origin time. 2024-09-08T21:41:59 And that one is from February of 2024. 2024-09-08T21:43:51 should I add an index now so I don't have to wait on every SELECT ? I can't find how safe it is live .. 2024-09-08T21:44:41 can you show create table transactions; first so that we see the existing indexes? 2024-09-08T21:45:04 in the upstream ipsilon test suite they seem to do "CREATE INDEX idx_transactions_uuid ON transactions (uuid);" 2024-09-08T21:45:07 sure 2024-09-08T21:45:56 https://paste.opensuse.org/pastes/02e8320d7251 2024-09-08T21:49:13 ok, so there are keys on uuid and name, but not on value - but https://paste.opensuse.org/pastes/92a154878459 shows that a key on value would be valueable (sorry for the pun) 2024-09-08T21:49:58 now the question is if this should be only on value, or include other columns 2024-09-08T21:50:20 Isn't that type of declaration basically saying that the unique primary key is uuid+name? ie, you couldn't have two rows with the same uuid and name but a different value? 2024-09-08T21:50:30 nice pun 2024-09-08T21:50:46 So you get one uuid+cookie (for example) 2024-09-08T21:50:54 right 2024-09-08T21:51:10 Been a long time since my college database class :D 2024-09-08T21:51:14 (and while it's understandable, it's another bad idea from the database design POV) 2024-09-08T21:52:34 hm ok so if we're not certain I rather not break it more .. here is the origintime query https://paste.opensuse.org/pastes/8ed9742bd6ac 2024-09-08T21:52:35 funnily the idx_transactions_uid key on uuid is basically useless because it's a subset of the primary key, so any query that uses it could also use the primary key 2024-09-08T21:52:59 interesting 2024-09-08T21:53:28 but back to the missing key for value 2024-09-08T21:53:41 basically there are two options that make sense: 2024-09-08T21:54:09 - add a key only for value (including name isn't really helpful because there can only be 10 different names) 2024-09-08T21:55:37 - add a key for value, name, uuid (in this order) so that the query can be answered by the index without looking at the actual table (might be faster in theory, but with 3 text columns, the index becomes much bigger and might kill the theoretic advantage) 2024-09-08T21:56:22 my choice would be to add a key only for value 2024-09-08T21:58:35 I see, how risky do you deem this? 2024-09-08T21:58:59 if it's risky I rather make a task to do offline it as part of work maintenance 2024-09-08T21:59:02 Here's an idea, maybe. 2024-09-08T21:59:56 adding an index is usually very low risk 2024-09-08T22:00:52 the issue here is the size of the table, so it will take some time 2024-09-08T22:01:08 The only risk I'd see is it requires additional disk space, probably not as much as we have free, but if the issue ultimately is that this background cleanup process is consuming the disk space, that will happen marginally (probably) more quickly. 2024-09-08T22:01:28 during that time, logins might be delayed or even fail _if_ the implementation is blocking (but I'd hope it isn't) 2024-09-08T22:01:44 the filling up disk space is on /, but the data directory is on a different disk which has ~200G free 2024-09-08T22:02:02 Gotcha. 2024-09-08T22:02:25 cboltz: I see, I'm not sure seeing this software implemented I want to go for hope :) 2024-09-08T22:03:32 thanks for the analysis, I'll note to do it properly, then I can also backup first 2024-09-08T22:04:28 in this case, it's implementation in MariaDB 2024-09-08T22:07:48 @acidsys 2024-09-08T22:08:01 Do you see anything in any logs related to transaction ID "4b8362c6-0bc0-43fb-b803-fb0a365d73d7"? 2024-09-08T22:09:58 yep 2024-09-08T22:10:28 That was a failure that happened while we were talking a little earlier, after debug was turned on 2024-09-08T22:10:40 I think while you were running one of the queries. So maybe we have what we need? 2024-09-08T22:11:31 that's the 20:56 one I think I checked above 2024-09-08T22:12:06 21:02 I don't find any errors in your messages.. only some info that you log in to the app discourse.opensuse.org 2024-09-08T22:13:31 I can paste it if you want but it contains your PII 2024-09-08T22:14:21 Ah, the duplicate entry message? (from 2:02 my time) 2024-09-08T22:15:09 no, that one I found upon searching for src.opensuse.org, but it is from hours earlier 2024-09-08T22:15:20 Ah, that's why the IDs don't show up. 2024-09-08T22:15:25 Er, match. 2024-09-08T22:21:23 Oh, the ID is for the one I did around that time, not from earlier in the day 2024-09-08T22:21:52 Looks like the timestamp is 21:26:07 GMT 2024-09-08T22:25:03 Oh, wait, no, that's a cookie expiration. The actual timestamp is 20:56:07. So yeah, that's the same one 2024-09-08T23:20:23 acidsys: I still randomly can't access gitlab because of the A records existing. 2024-09-08T23:20:41 Oh maybe it's not that 2024-09-08T23:20:43 Hmmm 2024-09-08T23:20:46 Firefox loves to not connect to it. 2024-09-08T23:21:07 there is no A record in our DNS for it. 2024-09-08T23:21:30 I use firefox 2024-09-08T23:21:47 Hmmm 2024-09-08T23:21:55 I have dns over https disabled 2024-09-08T23:22:00 Need to dig into that again then. 2024-09-08T23:22:02 I have that disabled too 2024-09-08T23:24:22 you can check in about:networking#dnslookuptool 2024-09-08T23:24:44