| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-07-27 | |||
| 10:22:13 | sean-k-mooney | always in the lvm job for some reason and not others | |
| 10:23:32 | sean-k-mooney | looking at the test im wondering if this could be a test bug or not | |
| 10:23:34 | sean-k-mooney | https://github.com/openstack/tempest/blob/04cb0adc822ffea6c7bfccce8fa08b03739894b7/tempest/api/compute/servers/test_server_actions.py#L713 | |
| 10:23:54 | sean-k-mooney | im wondering if we are perhaps stoping the server before the guest writes to the console? | |
| 10:27:21 | sean-k-mooney | https://opensearch.logs.openstack.org/_dashboards/app/discover?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-30d,to:now))&_a=(columns:!(hosts_region,project,build_name),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'94869730-aea8-11ec-9e6a-83741af3fdcd',key:project,negate:!f,params:(query:openstack%2Fnova),ty | |
| 10:27:23 | sean-k-mooney | pe:phrase),query:(match_phrase:(project:openstack%2Fnova)))),index:'94869730-aea8-11ec-9e6a-83741af3fdcd',interval:d,query:(language:kuery,query:'message:%22Console%20output%20was%20empty.%22'),sort:!()) | |
| 10:27:31 | sean-k-mooney | hum let me get a short link | |
| 10:28:31 | sean-k-mooney | https://tinyurl.com/kvcc9289 | |
| 10:29:06 | sean-k-mooney | 132 failures in the last 3 days | |
| 10:36:32 | bauzas | sean-k-mooney: ack, ok | |
| 10:37:19 | bauzas | sorry, was on meeting | |
| 10:37:32 | sean-k-mooney | no worries just filed https://bugs.launchpad.net/nova/+bug/2028851 | |
| 10:38:06 | sean-k-mooney | im going ot add a wiat for sshabel before the server stop in tempest and see if that helps | |
| 10:38:39 | bauzas | sean-k-mooney: it looks to me it regressed two days before https://zuul.openstack.org/builds?job_name=nova-lvm&skip=0 | |
| 10:39:00 | bauzas | last successful run was 2023-07-24 09:18:03 | |
| 10:39:01 | sean-k-mooney | ya it stated on the 24th | |
| 10:39:22 | sean-k-mooney | im not really sure what could have merged then | |
| 10:39:59 | bauzas | looking at gerrit | |
| 10:40:08 | bauzas | it doesn't sound a race, rather a regression | |
| 10:40:10 | sean-k-mooney | https://github.com/openstack/nova/commits/master | |
| 10:40:34 | sean-k-mooney | we had one commit on the 21st and then the 25th | |
| 10:40:48 | sean-k-mooney | so i dont see anything in nova that aligns | |
| 10:42:39 | bauzas | yeah | |
| 10:43:51 | bauzas | sean-k-mooney: but tempest has suspects https://github.com/openstack/tempest/commits/04cb0adc822ffea6c7bfccce8fa08b03739894b7 | |
| 10:44:16 | sean-k-mooney | ya i looked at the optimize one | |
| 10:44:26 | sean-k-mooney | i dont think that is touching the relevent test | |
| 10:45:11 | bauzas | https://github.com/openstack/tempest/commit/eea2c1cfac1e5d240cad4f8be68cff7d72f220a8 | |
| 10:45:31 | sean-k-mooney | oh the merge commit | |
| 10:45:37 | bauzas | we have a problem with server actions but this patch touches this | |
| 10:47:37 | sean-k-mooney | yep it moved the test | |
| 10:48:36 | sean-k-mooney | this is the orginal commit https://github.com/openstack/tempest/commit/1643536c9f7407dceccfaa18a2dd95156de1419c looks like dan wrote it | |
| 10:49:02 | bauzas | probably an unnotified merge conflict when rebasing | |
| 10:50:20 | sean-k-mooney | i dont think so | |
| 10:50:38 | sean-k-mooney | well maybe so what dan did was split up the large class into 3 | |
| 10:50:46 | sean-k-mooney | so that we got bettter parallisum | |
| 10:51:12 | sean-k-mooney | reducing the over all time of the tests form 15 min to 5 on there system | |
| 10:51:21 | sean-k-mooney | but that might mean we have more load | |
| 10:51:34 | sean-k-mooney | and the vm boots might now be slower | |
| 10:53:42 | bauzas | yeah sounds plausible | |
| 10:54:06 | sean-k-mooney | im just going to change server = self.create_test_server(wait_until='ACTIVE') | |
| 10:54:08 | sean-k-mooney | to server = self.create_test_server(wait_until='SSHABLE') | |
| 10:56:09 | bauzas | shoot | |
| 10:56:20 | bauzas | and depends-on it with a nova dnm please | |
| 10:56:31 | sean-k-mooney | yep | |
| 10:57:37 | sean-k-mooney | now i want one :) | |
| 10:57:45 | sean-k-mooney | enjoy | |
| 11:03:46 | opendevreview | sean mooney proposed openstack/nova master: use disk caching to hide slow cinder performance https://review.opendev.org/c/openstack/nova/+/889383 | |
| 11:04:40 | sean-k-mooney | ill use my diskcache mode change to test it instead ^ that way we can test both and save ci time | |
| 12:56:42 | opendevreview | alecorps proposed openstack/nova-specs master: new spec: support of vnc console for mks https://review.opendev.org/c/openstack/nova-specs/+/889857 | |
| 13:29:45 | dansmith | sean-k-mooney: if you could use my nick so I get a notify I'd appreciate it | |
| 13:29:52 | dansmith | I can't notify on "dan" because it's far too common | |
| 13:30:03 | dansmith | sean-k-mooney: so which test are you changing from active to sshable? | |
| 13:30:29 | dansmith | the split patch literally just moved code, so it shouldn't affect anything | |
| 13:30:59 | dansmith | it will increase parallelism, but we usually are running that test when the workers are fully loaded with tests, not after they've run out of work, | |
| 13:31:10 | dansmith | so I wouldn't expect it to really change the profile much | |
| 13:35:18 | dansmith | bauzas: can you please get this queued up? you've already +2d the follow-up to address your feedback: https://review.opendev.org/c/openstack/nova/+/884752/2 | |
| 13:35:30 | bauzas | dansmith: cool | |
| 13:35:37 | dansmith | I've been rechecking these for three weeks and I'm getting close, so I want to be able to land this when the time comes | |
| 13:35:51 | bauzas | dansmith: as we discussed with sean-k-mooney this morning, we have a job issue | |
| 13:36:37 | dansmith | I realize that, I still would like to have it ready to go, the patch below it is still waiting so +W isn't going to send it | |
| 13:37:02 | sean-k-mooney | i dont think we should slow down reviewing bacause of the job issue | |
| 13:37:15 | dansmith | agree | |
| 13:37:43 | dansmith | sean-k-mooney: I see the console_output thing, that's because it's being requested as sshable and doesn't have the validation things it needs | |
| 13:38:00 | dansmith | sean-k-mooney: not sure how that changed for that test, but I can fix it easily | |
| 13:38:31 | dansmith | sean-k-mooney: did you open a bug for it? | |
| 13:38:39 | sean-k-mooney | i fixed the validation thing | |
| 13:38:45 | sean-k-mooney | i opened a bug | |
| 13:38:52 | sean-k-mooney | and i have a ptach in ci right now | |
| 13:39:01 | sean-k-mooney | its almost completed running | |
| 13:39:02 | dansmith | link? | |
| 13:39:15 | sean-k-mooney | https://review.opendev.org/c/openstack/tempest/+/889847 | |
| 13:39:24 | sean-k-mooney | https://launchpad.net/bugs/2028851 | |
| 13:39:33 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/889383 | |
| 13:39:44 | sean-k-mooney | i used my disk caching patch to run it with a depends on | |
| 13:39:52 | sean-k-mooney | https://zuul.openstack.org/stream/9890c9b53059410eabc0c61df0376217?logfile=console.log | |
| 13:40:02 | sean-k-mooney | that is the nova-lvm job i have it open on my other screen | |
| 13:40:03 | dansmith | oh I was looking at this: https://8d44202ac4ef26a68d85-49968699300828e6c9b78fd54dff75ef.ssl.cf1.rackcdn.com/889383/4/check/nova-lvm/368f504/testr_results.html | |
| 13:40:06 | dansmith | the in_shutoff test fail | |
| 13:40:14 | dansmith | where it claims it doesn't have validation_resourecs | |
| 13:40:36 | dansmith | sorry, no the unlimited_size one | |
| 13:40:45 | dansmith | well, both | |
| 13:40:55 | dansmith | ah, that's yours | |
| 13:40:56 | sean-k-mooney | ya in a previous revsion i enabled sshable | |
| 13:41:01 | sean-k-mooney | but not the validation resouces | |
| 13:41:07 | dansmith | ah, okay cool | |
| 13:42:15 | dansmith | okay so yep, increased load making us lose that race makes sense | |
| 13:42:37 | dansmith | it's definitely not 100% fail | |
| 13:43:07 | sean-k-mooney | its only happening in the lvm job and i think that uses lvm on a loopback file so i suspect the disk io is worse there then other jobs | |
| 13:43:13 | dansmith | because we've merged stuff in the last couple of days | |
| 13:44:09 | dansmith | er | |
| 13:44:24 | dansmith | I actually don't see that on those patches.. is that in experimental or something? | |
| 13:44:40 | sean-k-mooney | no https://github.com/openstack/nova/blob/master/.zuul.yaml#L230-L244 | |
| 13:44:48 | sean-k-mooney | but it does not run on all changes | |
| 13:45:13 | sean-k-mooney | it only runs on change to libvirt driver | |
| 13:45:15 | dansmith | ah, only on libvirt I see | |
| 13:45:16 | dansmith | okay | |
| 13:45:25 | dansmith | well in that case we *definitely* shouldn't be holding up review ;P | |
| 13:45:38 | bauzas | sorry, on an internal meeting now | |
| 13:46:22 | bauzas | dansmith: yeah, I haven't asked to hold reviews :) | |
| 13:46:45 | dansmith | bauzas: sounded like you did, but okay :) | |