| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-01-17 | |||
| 12:35:37 | sean-k-mooney | then use rescue mode on both vms to boot the vms with a different imave with a disk imagain tool | |
| 12:36:20 | sean-k-mooney | and then use your tool of choice to copy the data form 1 vm to the other | |
| 12:36:35 | sean-k-mooney | then unrescue to reboot from the vm root disk | |
| 12:36:48 | jawad_axd | Okay.. | |
| 12:36:57 | jawad_axd | I am going to try it.Thank you! | |
| 12:38:18 | sean-k-mooney | there are some third part disater recovery tools that claim to be able to do this for you | |
| 12:38:25 | sean-k-mooney | but i dont really recommend using them | |
| 12:38:49 | sean-k-mooney | mainly because they tend to mess with the nova db in ways we would not support | |
| 12:39:13 | jawad_axd | ok | |
| 13:50:50 | gibi | efried: left comments in the first provider config patch: https://review.opendev.org/#/c/673341 I think my comments about the schema would be interesting to you too | |
| 13:59:17 | gibi | dustinc|PTO: ^^ | |
| 14:01:41 | openstackgerrit | Vladyslav Drok proposed openstack/nova master: Make removal of host from aggregate consistent https://review.opendev.org/696392 | |
| 14:08:05 | dustinc|PTO | Thanks gibi, I read through your comments but won’t be able to follow up until Tuesday. | |
| 14:09:00 | gibi | dustinc|PTO: no worries I also need to read your other patches in the series | |
| 14:09:25 | gibi | dustinc|PTO: have a nice PTO | |
| 14:09:47 | dustinc|PTO | Thanks :) | |
| 14:31:51 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/stein: DNM: try to trace bug 1859766 https://review.opendev.org/702700 | |
| 14:31:51 | openstack | bug 1859766 in OpenStack Compute (nova) stein "functional tests intermittently fails with "ReadOnlyFieldError: Cannot modify readonly field uuid"" [Undecided,New] https://launchpad.net/bugs/1859766 - Assigned to Balazs Gibizer (balazs-gibizer) | |
| 14:43:17 | lyarwood | Did anyone get to the bottom of `mysql: command not found` on the subnodes in the LM jobs? | |
| 14:46:18 | gibi | lyarwood: based on the bug https://bugs.launchpad.net/nova/+bug/1860021 we still have the problem | |
| 14:46:18 | openstack | Launchpad bug 1860021 in devstack "nova-live-migration fails 100% with "mysql: command not found" on subnode" [Undecided,In progress] - Assigned to Radosław Piliszek (yoctozepto) | |
| 14:46:56 | lyarwood | gibi: thanks, I couldn't find that for some reason. | |
| 14:47:02 | lyarwood | and yeah glance on the subnode is the issue | |
| 14:56:03 | efried | thanks gibi | |
| 14:56:36 | lyarwood | g-api on computes is just weird tbh, now if I could just figure out where the hell that's set | |
| 15:00:56 | lyarwood | ah it is devstack-gate | |
| 15:02:40 | efried | lyarwood: If we turn that off, won't it break grenade? | |
| 15:02:53 | efried | sean-k-mooney pointed out the commit from mriedem where that was introduced. | |
| 15:05:41 | lyarwood | efried: I'd have to check that commit for context, is there a rolling g-api upgrade test in grenade? | |
| 15:06:14 | efried | I have no earthly idea. Grenade has always been a swirling mystery to me. | |
| 15:07:00 | efried | lyarwood: https://github.com/openstack/devstack-gate/commit/183a58eab185c2e6f48afb4d99417d49e88c0f5a#diff-987768d02fbf43ebb74ad9a359222903R27 | |
| 15:07:03 | lyarwood | Well that's two of us | |
| 15:08:12 | efried | Shooting in the dark, we can proposed a d-g patch to remove g-api from that list, and a nova patch to depend on it, and see what goes kablooey. | |
| 15:08:57 | lyarwood | efried: yup just doing that now | |
| 15:09:05 | lyarwood | efried: I can't think why that would break anything in grenade tbh | |
| 15:09:08 | efried | oh, k, /me stops doing that. | |
| 15:09:11 | lyarwood | efried: but if it does I'll dig into it | |
| 15:09:22 | efried | great, thank you lyarwood | |
| 15:10:37 | openstackgerrit | Lee Yarwood proposed openstack/nova master: DNM Test the removal of g-api from subnodes https://review.opendev.org/703100 | |
| 15:10:40 | dansmith | has anyone asked glance if they're doing that on purpose? (or is it just our jobs?) | |
| 15:11:04 | dansmith | because they've been working on some multi-worker multi-store stuff lately which likely needs g-api on different nodes to accomplish | |
| 15:12:59 | sean-k-mooney | running the glacne api on multiple nodes does not require them to initalise the glance db multiple times | |
| 15:13:33 | lyarwood | right we aren't talking about that bug here, just if running g-api on a subnode is valid. | |
| 15:13:34 | sean-k-mooney | so if they are which would be resonable devstack need to be updated so that we can make sure the db is only setup on the contoler | |
| 15:13:48 | sean-k-mooney | lyarwood: sure you can do that | |
| 15:14:00 | sean-k-mooney | normally we would not need too but you can | |
| 15:14:18 | melwitt | I can't find where in the chat log someone mentioned it would be needed for grenade | |
| 15:14:22 | sean-k-mooney | i have seen it done on small 1-3 node deployments | |
| 15:14:41 | dansmith | specifically for edge it's a thing | |
| 15:14:48 | sean-k-mooney | where all the contoler were also computes and they split the contol plane across them | |
| 15:15:16 | melwitt | http://eavesdrop.openstack.org/irclogs/%23openstack-qa/%23openstack-qa.2020-01-16.log.html#t2020-01-16T18:56:14 | |
| 15:15:44 | sean-k-mooney | melwitt: it might be need for grenade to test that we can upgrade nova independly of glance or something like that | |
| 15:15:45 | lyarwood | yeah of course, I guess devstack-gate just needs some additional subnode flavours if you will covering a pure compute subnode, mixed subnode with g-api etc | |
| 15:16:17 | sean-k-mooney | lyarwood: or we could move to zullv3 | |
| 15:16:22 | lyarwood | or that | |
| 15:16:46 | sean-k-mooney | althogh devstack would still need to have configuration for the db | |
| 15:16:59 | sean-k-mooney | also crap im late for the bug call | |
| 15:17:29 | lyarwood | sean-k-mooney: you're not missing much :) | |
| 15:17:52 | sean-k-mooney | my only bug was closed so i didnt have anything to talk about. | |
| 15:18:05 | sean-k-mooney | i just realise what time it was | |
| 15:18:25 | sean-k-mooney | backporting functional test is annoying | |
| 15:19:07 | sean-k-mooney | apparently we rewrite how the numa ones work each cycle | |
| 15:20:50 | melwitt | I'm working on testing out grenade with zuul v3 in nova (https://review.opendev.org/701478 depends-on https://review.opendev.org/548936 and https://review.opendev.org/703005 | |
| 15:21:44 | melwitt | not as a fix for the gate but to get away from the gzipped log files | |
| 15:22:11 | sean-k-mooney | did my change not work | |
| 15:22:20 | sean-k-mooney | i saw a ping about it last night | |
| 15:22:28 | sean-k-mooney | but i did not fully get the context | |
| 15:22:45 | sean-k-mooney | but yes i would like use to adopt the v3 approch if we can too | |
| 15:23:07 | gmann | sean-k-mooney: it fixed for zuulv3 jobs not legacy which are d-g dependents not on devstack roles | |
| 15:23:30 | melwitt | yep, what gmann said | |
| 15:23:49 | sean-k-mooney | ah ok | |
| 15:24:34 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/stein: Remove global state from the FakeDriver https://review.opendev.org/703102 | |
| 15:24:35 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/stein: Enhance service restart in functional env https://review.opendev.org/703103 | |
| 15:24:35 | melwitt | so I found that proposed grenade patch that moves it to v3 and it mostly works. just needs a few updates | |
| 15:25:14 | gmann | melwitt: yeah only remaining things there is that does not work for plugin case yet. which is something should be figured out before we merge that | |
| 15:25:21 | sean-k-mooney | melwitt: yep i have been trying to get peopel to review it ever few weeks for a while | |
| 15:25:35 | sean-k-mooney | gmann: there is one other case too | |
| 15:25:46 | melwitt | gmann: oh I see | |
| 15:26:12 | sean-k-mooney | gmann: it can be done as a follow up but you cant specify a different local_conf section per version | |
| 15:26:18 | sean-k-mooney | you can do it for localrc | |
| 15:26:27 | gmann | sean-k-mooney: i could not get chance to find gzip log place for legacy jobs. busy for tempest plugins py2-drop drama bug. | |
| 15:26:38 | sean-k-mooney | but localrc does not allow you to set the post-config updates | |
| 15:28:01 | gmann | sean-k-mooney: yeah that(different local_conf section) is true, i failed to do that while IPv6 enable for legacy jobs. | |
| 15:28:44 | gmann | but i am not sure why devstsack fix is wrong, waiting for frickler re-reply | |
| 15:34:08 | sean-k-mooney | gmann: we will need that to be able to test placement reshapes on upgrade in grenade | |
| 15:34:39 | sean-k-mooney | because we sill need to change form using vcpu_pin_set to cpu_share_set and cpu_dedicated_set | |
| 15:34:51 | sean-k-mooney | although we dont test that today so its not required to move | |
| 15:35:53 | sean-k-mooney | im sure there are other examples too but having the ablity to specifc new/old/shared local_conf sections in the zuul.yaml will be useful | |
| 15:38:25 | tosky | melwitt: "mostly" - for some reasons, after we switched to Python 3, several python 2 packages are installed anyway, including python-openstackclient | |
| 15:39:16 | tosky | melwitt: so whenever "openstack foo" is called, the py2 version is used; and when the sahara plugin (or heat plugin) tries creating resources with "openstack dosomething", its own osc plugin (py3 only) is not found | |
| 15:39:23 | gmann | sean-k-mooney: or we can do that via direct config change via roles. something to explore may be. | |
| 15:39:26 | tosky | I'm failing to understand where the issue is | |
| 15:40:06 | tosky | gmann, sean-k-mooney or the from-<foo> update plugin maybe | |
| 15:40:09 | melwitt | I see | |
| 15:40:37 | sean-k-mooney | gmann: we could but im not sure that would be better. if it requrie new roles to be written i think it would be worse the adding supprot for local.conf in grenade | |
| 15:40:49 | lyarwood | random question, what's the best way of checking the stability of a given job over the last x number of months? | |
| 15:41:19 | sean-k-mooney | tosky: it might be the fact that on older brances we install everything twice. once under py3 then again under py2 | |
| 15:41:26 | gmann | lyarwood: http://status.openstack.org/openstack-health/#/ | |
| 15:41:37 | sean-k-mooney | tosky: we only stopped doing that in ussuri | |
| 15:41:39 | gmann | you can filter the jobs and time period | |