| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-09-29 | |||
| 12:18:14 | mdbooth | Ok, I can confirm that the image the machine is created from already contains dirty devstack state | |
| 12:18:28 | mdbooth | Including the broken ProxyPass directives | |
| 12:19:08 | mdbooth | So unless we consider it a bug that devstack doesn't update that, I'm guessing this is my problem? | |
| 12:20:30 | gibi | mdbooth: so you have both wrong and bad ProxyPass lines in the config file? | |
| 12:20:41 | gibi | :D | |
| 12:20:41 | gibi | I mean wrong and good | |
| 12:21:11 | mdbooth | So *before* pulling and executing a patched devstack I already have bad config present | |
| 12:21:21 | mdbooth | And running the patched devstack doesn't fix it | |
| 12:21:58 | gibi | I do see multiple ProxyPass lines locally too, I guess those are from multiple unstack.sh / stack.sh runs in my case | |
| 12:22:04 | lyarwood | yeah that isn't a devstack bug | |
| 12:22:07 | lyarwood | ./clean.sh first | |
| 12:22:38 | gibi | hm, if clean.sh removes it then I agree it is not a bug (I'm lazy to always run clean.sh but I should) | |
| 12:23:03 | mdbooth | lyarwood: Or more likely summon the arcane wizards of CI and update the 'preinstalled' image! | |
| 12:23:22 | lyarwood | yeah or that if it's already baked into the image | |
| 12:23:24 | gibi | mdbooth: yeah, for CI I suggest to always start from a clean image :) | |
| 12:23:26 | lyarwood | that's odd tbh | |
| 12:24:05 | mdbooth | Apparently it saves a ton of time, although I haven't personally measured it. I'm about to measure it, though :) | |
| 12:24:44 | gibi | not too long ago dansmith added parallelism to devstack stack.sh run that helped a lot with runtime | |
| 12:27:51 | lyarwood | yeah that we use across most jobs now AFAIK | |
| 12:28:02 | mdbooth | lyarwood: Is there a flag? | |
| 12:29:16 | gibi | there should be | |
| 12:29:19 | gibi | but is on by default | |
| 12:29:27 | mdbooth | Ok, cool | |
| 12:29:57 | lyarwood | yeah it's there by default in master and xena | |
| 12:29:58 | gibi | btw, ./clean.sh deletes all the config from /etc/apache2/sites-enabled/ except glance :D | |
| 12:30:06 | mdbooth | Ah, looks like it's DEVSTACK_PARALLEL, and it's not in victoria | |
| 12:30:12 | mdbooth | Perhaps I'll update | |
| 12:30:34 | lyarwood | why are you deploying victoria btw? | |
| 12:31:02 | mdbooth | lyarwood: Because nobody changed it | |
| 12:31:18 | lyarwood | oh fun | |
| 12:31:34 | gibi | :) | |
| 12:32:22 | mdbooth | Honestly this CI system is awesome, I'm not complaining. It installs devstack in a VM in GCE and runs tests against it, and until we hit this it was rock solid. | |
| 12:33:40 | mdbooth | Ok, I'm going to run again from a clean image instead of the preinstalled image to see how long that takes. Then I'm going to do the same again but against Xena to measure again. | |
| 12:33:45 | mdbooth | Thanks for all the help! | |
| 12:34:19 | gibi | happy to help | |
| 12:36:25 | gmann | bauzas: other than neutron-grenade. nova-ceph-multistore is broken on stable (https://bugs.launchpad.net/devstack-plugin-ceph/+bug/1945358) fixes are ready to merger - https://review.opendev.org/q/topic:%22bug%252F1945358%22+(status:open%20OR%20status:merged) | |
| 12:36:37 | gmann | gibi: lyarwood checking comments on grenade fixes, | |
| 12:39:16 | gibi | gmann: in short we are questioning whether we need to turn on trunk testing on stable branches | |
| 12:41:15 | gmann | gibi: lyarwood ok, I will say we should as Tempest master is used to test the stable/ussuri -> master and if we do not enable then we skip the trunk test for no reason. it was just a miss in enabling the extension in stable branches. | |
| 12:41:39 | lyarwood | gm | |
| 12:41:41 | lyarwood | ops sorry | |
| 12:42:06 | lyarwood | gmann: Yeah I appreciate that but if wasn't tested until now it seems a little odd to add it in for non-master branches | |
| 12:42:29 | lyarwood | gmann: the missing coverage was mostly around live migration right? | |
| 12:42:37 | gmann | and we can see it is passing in nova grenade jobs in stable so we do not need anythings extra than grenade fix | |
| 12:43:04 | gmann | lyarwood: yeah, live migration trunk tests. those are only tests for trunk currently in tempest | |
| 12:43:06 | lyarwood | gmann: okay well if it's passing and the neutron folks are okay with helping with any fallout that might appear in the coming days then I guess we can go ahead | |
| 12:43:44 | gmann | lyarwood: yeah, if neutron team is not comfortable or confident then we can just keep it like that. | |
| 12:53:46 | gmann | lyarwood: whether we need it for Xena or not, I remember master testing patch passed only when I added xena fix as depends-on here https://review.opendev.org/c/openstack/nova/+/811118 | |
| 12:54:08 | gmann | I think i added due to zuul job inventory but let me debug that what exactly happing here | |
| 12:54:14 | lyarwood | okay if that's the case lets just merge both | |
| 13:08:52 | gmann | lyarwood: zuul inventory seems to take job definition from master only https://zuul.opendev.org/t/openstack/build/01f10e2cb162450e912551026dde8f85/log/zuul-info/inventory.yaml#294-304 | |
| 13:09:08 | gmann | let me remove the xena fix as depends-on and then we can get more clarity | |
| 13:09:45 | opendevreview | Ghanshyam proposed openstack/nova master: DNM testing grenade neutron-trunk fix https://review.opendev.org/c/openstack/nova/+/811118 | |
| 13:16:32 | lyarwood | gmann: kk | |
| 13:55:54 | bauzas | sorry folks, I had issues with my computer | |
| 13:56:08 | bauzas | gmann: thanks, unfortunately gtk | |
| 13:57:37 | bauzas | looks like it's limbo | |
| 13:58:31 | gmann | nova grenade master fix is in gate https://review.opendev.org/c/openstack/grenade/+/811117 | |
| 13:58:57 | gmann | for enabling testing on stable or not is different things and we can continue discussion. | |
| 14:01:40 | dansmith | bauzas: gibi looks like the nova-yoga-ptg etherpad has been emptied | |
| 14:01:51 | bauzas | ORLY ? | |
| 14:02:21 | gibi | shit, I see | |
| 14:02:52 | bauzas | dansmith: this was really done 3 mins before | |
| 14:02:56 | gibi | this is the last known state https://etherpad.opendev.org/p/nova-yoga-ptg/timeslider#8272 | |
| 14:03:14 | gibi | last known good state | |
| 14:03:16 | bauzas | yup | |
| 14:03:18 | dansmith | bauzas: okay I just loaded it and saw it empty | |
| 14:03:24 | bauzas | dansmith: me too | |
| 14:03:36 | bauzas | it's just, something happened between 8272 and 8273 | |
| 14:03:43 | dansmith | certainly you can restore a rev right? | |
| 14:03:58 | gibi | the content yes, the coloring I think no | |
| 14:04:11 | bauzas | I wonder | |
| 14:04:18 | bauzas | lemme ask infra | |
| 14:04:20 | gibi | sure | |
| 14:04:21 | dansmith | hmm, I thought there was some way | |
| 14:04:31 | gibi | what I did before is that I exported the old state and copied that back | |
| 14:05:37 | bauzas | fungi is looking at restoring | |
| 14:05:41 | gibi | ack | |
| 14:05:43 | bauzas | folks, don't touch now | |
| 14:05:49 | dansmith | sweet | |
| 14:06:01 | bauzas | dansmith: thanks for having identified this | |
| 14:06:39 | lyarwood | ah ffs I think that was me | |
| 14:06:59 | lyarwood | I was editing and my dock crashed | |
| 14:07:12 | lyarwood | and then I couldn't reconnect to the pad | |
| 14:08:13 | lyarwood | yeah my items were the last on there | |
| 14:08:28 | lyarwood | https://etherpad.opendev.org/p/nova-yoga-ptg/timeslider#8272 | |
| 14:09:07 | lyarwood | not entirely sure how my laptop dock (and thus network) crashing caused this tbh | |
| 14:09:58 | gibi | an interesting untested edge case in the etherpad code :) | |
| 14:13:08 | fungi | bauzas: gibi: dansmith: i've rolled it back to revision 8272 | |
| 14:13:19 | fungi | lyarwood: ^ | |
| 14:13:20 | dansmith | looks good, thanks fungi ! | |
| 14:13:23 | bauzas | fungi: with all my love | |
| 14:13:36 | fungi | cool, just making sure it's looking like you needed | |
| 14:13:50 | lyarwood | thanks for that and apologies all | |
| 14:13:56 | fungi | we do also back up the db behind it daily, worst case | |
| 14:14:04 | bauzas | fungi: it is, all the colors | |
| 14:14:07 | gibi | fungi: awesome thanks | |
| 14:14:10 | fungi | no problem, glad we could recover it | |
| 14:14:28 | bauzas | and yeah, interesting edge case | |
| 14:14:47 | bauzas | a docking issue with a network problem swallows an etherpad | |