| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-01-17 | |||
| 12:23:54 | yoctozepto | so it's a bug in libvirt losing hvm | |
| 12:24:17 | admin0 | thanks yoctozepto, sean-k-mooney .. can i mention a thank you when i put put this article on my openstackfaq.com site | |
| 12:24:19 | sean-k-mooney | admin0: you might want to take this to #openstack by the way | |
| 12:24:33 | admin0 | i did started on kolla aio https://www.openstackfaq.com/openstack-kolla-all-in-one/ | |
| 12:24:45 | sean-k-mooney | admin0: sure if you want | |
| 12:25:03 | yoctozepto | admin0: /me appreciates appreciation | |
| 12:25:03 | sean-k-mooney | jawad_axd: there is noe cross cloud migration support native to nova | |
| 12:25:30 | sean-k-mooney | jawad_axd: so your best bet would be to do the backup from within the vm. | |
| 12:25:50 | sean-k-mooney | e.g. snapshot root disk | |
| 12:26:11 | jawad_axd | @sean-k-mooney : Not migration, just want to take snapshot of instance with ephemeral disk? | |
| 12:26:11 | sean-k-mooney | create a vm on the ohter cloud by downloand an uploadign the snap shot. | |
| 12:26:28 | sean-k-mooney | boot it in rescue mode and then rsync the contentes fo the ephemeral volumes | |
| 12:26:49 | sean-k-mooney | you can take a snapshot but it will only snapshot the root disk | |
| 12:26:49 | admin0 | jawad_axd, or you can just boot 2 vms, rsync the data you need | |
| 12:27:05 | sean-k-mooney | that is basicly what i was saying | |
| 12:27:36 | yoctozepto | (this is the point where one suggests using dd over socat) | |
| 12:27:47 | jawad_axd | Snapshot doest work with ephemeral disk[E.g flavor has root disk and ephemral disk], while snapshot works with just root disk when in flavor ephemeral disk=0 . | |
| 12:28:16 | jawad_axd | ah ok | |
| 12:28:22 | sean-k-mooney | really it should still work with the root disk. at least i taught it did | |
| 12:28:33 | sean-k-mooney | i just wont snapshot the ephemeral disks | |
| 12:28:58 | jawad_axd | Its not even snapshotting the root disk. | |
| 12:29:01 | sean-k-mooney | ephemeral disk are intened for fast local storage by the way. not data persitency | |
| 12:29:17 | sean-k-mooney | i see | |
| 12:29:46 | sean-k-mooney | that should be filed as a bug. thre may be a reson for that but i dont know of one off the top of my head | |
| 12:30:04 | sean-k-mooney | then what i would do is as follows | |
| 12:30:18 | sean-k-mooney | boot a second vm on the new cloud | |
| 12:30:29 | sean-k-mooney | boot both vms in rescue mode | |
| 12:30:53 | sean-k-mooney | then use dd/rsync/clonzilla to copy all disks between the two | |
| 12:30:59 | sean-k-mooney | then unrescue the new vm | |
| 12:31:48 | sean-k-mooney | if you are moving to ceph i would also recomend replaceign the epmeeral disk with cinder volumens | |
| 12:32:59 | sean-k-mooney | if your user have permission on both clouds then this is something they could do themsese if its a private cloud followign instuctions | |
| 12:33:37 | jawad_axd | Ok, I am trying to understand your instructions. | |
| 12:34:36 | sean-k-mooney | basically on the new cloud create a vm with X root disk and n cinder volumns to repalce the ephemeral disks | |
| 12:35:37 | sean-k-mooney | then use rescue mode on both vms to boot the vms with a different imave with a disk imagain tool | |
| 12:36:20 | sean-k-mooney | and then use your tool of choice to copy the data form 1 vm to the other | |
| 12:36:35 | sean-k-mooney | then unrescue to reboot from the vm root disk | |
| 12:36:48 | jawad_axd | Okay.. | |
| 12:36:57 | jawad_axd | I am going to try it.Thank you! | |
| 12:38:18 | sean-k-mooney | there are some third part disater recovery tools that claim to be able to do this for you | |
| 12:38:25 | sean-k-mooney | but i dont really recommend using them | |
| 12:38:49 | sean-k-mooney | mainly because they tend to mess with the nova db in ways we would not support | |
| 12:39:13 | jawad_axd | ok | |
| 13:50:50 | gibi | efried: left comments in the first provider config patch: https://review.opendev.org/#/c/673341 I think my comments about the schema would be interesting to you too | |
| 13:59:17 | gibi | dustinc|PTO: ^^ | |
| 14:01:41 | openstackgerrit | Vladyslav Drok proposed openstack/nova master: Make removal of host from aggregate consistent https://review.opendev.org/696392 | |
| 14:08:05 | dustinc|PTO | Thanks gibi, I read through your comments but won’t be able to follow up until Tuesday. | |
| 14:09:00 | gibi | dustinc|PTO: no worries I also need to read your other patches in the series | |
| 14:09:25 | gibi | dustinc|PTO: have a nice PTO | |
| 14:09:47 | dustinc|PTO | Thanks :) | |
| 14:31:51 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/stein: DNM: try to trace bug 1859766 https://review.opendev.org/702700 | |
| 14:31:51 | openstack | bug 1859766 in OpenStack Compute (nova) stein "functional tests intermittently fails with "ReadOnlyFieldError: Cannot modify readonly field uuid"" [Undecided,New] https://launchpad.net/bugs/1859766 - Assigned to Balazs Gibizer (balazs-gibizer) | |
| 14:43:17 | lyarwood | Did anyone get to the bottom of `mysql: command not found` on the subnodes in the LM jobs? | |
| 14:46:18 | gibi | lyarwood: based on the bug https://bugs.launchpad.net/nova/+bug/1860021 we still have the problem | |
| 14:46:18 | openstack | Launchpad bug 1860021 in devstack "nova-live-migration fails 100% with "mysql: command not found" on subnode" [Undecided,In progress] - Assigned to Radosław Piliszek (yoctozepto) | |
| 14:46:56 | lyarwood | gibi: thanks, I couldn't find that for some reason. | |
| 14:47:02 | lyarwood | and yeah glance on the subnode is the issue | |
| 14:56:03 | efried | thanks gibi | |
| 14:56:36 | lyarwood | g-api on computes is just weird tbh, now if I could just figure out where the hell that's set | |
| 15:00:56 | lyarwood | ah it is devstack-gate | |
| 15:02:40 | efried | lyarwood: If we turn that off, won't it break grenade? | |
| 15:02:53 | efried | sean-k-mooney pointed out the commit from mriedem where that was introduced. | |
| 15:05:41 | lyarwood | efried: I'd have to check that commit for context, is there a rolling g-api upgrade test in grenade? | |
| 15:06:14 | efried | I have no earthly idea. Grenade has always been a swirling mystery to me. | |
| 15:07:00 | efried | lyarwood: https://github.com/openstack/devstack-gate/commit/183a58eab185c2e6f48afb4d99417d49e88c0f5a#diff-987768d02fbf43ebb74ad9a359222903R27 | |
| 15:07:03 | lyarwood | Well that's two of us | |
| 15:08:12 | efried | Shooting in the dark, we can proposed a d-g patch to remove g-api from that list, and a nova patch to depend on it, and see what goes kablooey. | |
| 15:08:57 | lyarwood | efried: yup just doing that now | |
| 15:09:05 | lyarwood | efried: I can't think why that would break anything in grenade tbh | |
| 15:09:08 | efried | oh, k, /me stops doing that. | |
| 15:09:11 | lyarwood | efried: but if it does I'll dig into it | |
| 15:09:22 | efried | great, thank you lyarwood | |
| 15:10:37 | openstackgerrit | Lee Yarwood proposed openstack/nova master: DNM Test the removal of g-api from subnodes https://review.opendev.org/703100 | |
| 15:10:40 | dansmith | has anyone asked glance if they're doing that on purpose? (or is it just our jobs?) | |
| 15:11:04 | dansmith | because they've been working on some multi-worker multi-store stuff lately which likely needs g-api on different nodes to accomplish | |
| 15:12:59 | sean-k-mooney | running the glacne api on multiple nodes does not require them to initalise the glance db multiple times | |
| 15:13:33 | lyarwood | right we aren't talking about that bug here, just if running g-api on a subnode is valid. | |
| 15:13:34 | sean-k-mooney | so if they are which would be resonable devstack need to be updated so that we can make sure the db is only setup on the contoler | |
| 15:13:48 | sean-k-mooney | lyarwood: sure you can do that | |
| 15:14:00 | sean-k-mooney | normally we would not need too but you can | |
| 15:14:18 | melwitt | I can't find where in the chat log someone mentioned it would be needed for grenade | |
| 15:14:22 | sean-k-mooney | i have seen it done on small 1-3 node deployments | |
| 15:14:41 | dansmith | specifically for edge it's a thing | |
| 15:14:48 | sean-k-mooney | where all the contoler were also computes and they split the contol plane across them | |
| 15:15:16 | melwitt | http://eavesdrop.openstack.org/irclogs/%23openstack-qa/%23openstack-qa.2020-01-16.log.html#t2020-01-16T18:56:14 | |
| 15:15:44 | sean-k-mooney | melwitt: it might be need for grenade to test that we can upgrade nova independly of glance or something like that | |
| 15:15:45 | lyarwood | yeah of course, I guess devstack-gate just needs some additional subnode flavours if you will covering a pure compute subnode, mixed subnode with g-api etc | |
| 15:16:17 | sean-k-mooney | lyarwood: or we could move to zullv3 | |
| 15:16:22 | lyarwood | or that | |
| 15:16:46 | sean-k-mooney | althogh devstack would still need to have configuration for the db | |
| 15:16:59 | sean-k-mooney | also crap im late for the bug call | |
| 15:17:29 | lyarwood | sean-k-mooney: you're not missing much :) | |
| 15:17:52 | sean-k-mooney | my only bug was closed so i didnt have anything to talk about. | |
| 15:18:05 | sean-k-mooney | i just realise what time it was | |
| 15:18:25 | sean-k-mooney | backporting functional test is annoying | |
| 15:19:07 | sean-k-mooney | apparently we rewrite how the numa ones work each cycle | |
| 15:20:50 | melwitt | I'm working on testing out grenade with zuul v3 in nova (https://review.opendev.org/701478 depends-on https://review.opendev.org/548936 and https://review.opendev.org/703005 | |
| 15:21:44 | melwitt | not as a fix for the gate but to get away from the gzipped log files | |
| 15:22:11 | sean-k-mooney | did my change not work | |
| 15:22:20 | sean-k-mooney | i saw a ping about it last night | |
| 15:22:28 | sean-k-mooney | but i did not fully get the context | |