| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-01-05 | |||
| 17:51:54 | leakypipes | always remember, all of OpenStack can be rewritten entirely in SQL. | |
| 17:52:07 | cdent | as well as rewritten without any | |
| 17:52:08 | leakypipes | or Golang. | |
| 17:52:14 | sean-k-mooney | leakypipes: you could tripple your consluting fee to debug it if you wrote it lol | |
| 17:52:29 | leakypipes | sean-k-mooney: zero times three is zero. :) | |
| 17:54:16 | SamYaple | leakypipes: you mispelled rust | |
| 17:54:21 | sean-k-mooney | you know if you joke about rewriting openstack in golang 3 times it might summon the Ciao gods... | |
| 17:54:26 | leakypipes | SamYaple: :) | |
| 17:54:42 | leakypipes | SamYaple: you misspelled misspelled. | |
| 17:54:51 | SamYaple | i speel with the best of em | |
| 17:54:51 | leakypipes | :P | |
| 17:55:07 | leakypipes | sean-k-mooney: Beetlejuice! | |
| 17:55:38 | SamYaple | i did start on a libvirt replacement in rust (not 100% replacement, just spawn/monitor qemu process replacement) | |
| 17:55:50 | SamYaple | rust is suuuuuuuper ugly syntax wise | |
| 17:56:01 | sean-k-mooney | SamYaple: danial berage is considing that too. or golang | |
| 17:56:19 | SamYaple | im pretty sure i pushed *something* to git about it | |
| 17:56:29 | SamYaple | i had it spawning and interfacing with the monitor socket | |
| 17:56:52 | SamYaple | https://github.com/SamYaple/qemuwatch | |
| 17:57:03 | SamYaple | sean-k-mooney: i would love to not have to use libvirt anymore | |
| 17:57:13 | sean-k-mooney | SamYaple: there was a faily active thread on the libvirt mailing list back in december not sure if they reached a decision or not | |
| 17:57:37 | leakypipes | SamYaple: libvirt is a fine library. except for the XML. :) | |
| 17:57:50 | SamYaple | leakypipes: and the 500MB of deps needed to compile it | |
| 17:58:02 | leakypipes | SamYaple: the XML deps you mean, yeah? | |
| 17:58:21 | SamYaple | probably. i never dug into why libvirt was such a pita to build from source | |
| 17:58:24 | SamYaple | probably xml :) | |
| 17:59:33 | SamYaple | in truth, i was writing the above to spawn vms in kubernetes, libvirt is a bit overkill for that scenario (container per vm) | |
| 17:59:53 | SamYaple | but i stopped working on it because there are 1001 projects doing that right now and i prefer to get a more openstack native solution going | |
| 18:00:20 | sean-k-mooney | i havent done anything to maintain this since i gave it rodolfo to upstream but this made it less of a pain https://github.com/openstack/devstack-plugin-libvirt-qemu | |
| 18:01:06 | sean-k-mooney | i mean you are replaceing the pain of building libvirt from source with using devstack but at least you get to chose your poision | |
| 18:02:48 | SamYaple | i just want to use the native qemu interface more directly. live differential backups. add/remove hardware. better metrics. that kind of thing | |
| 18:02:59 | SamYaple | but libvirt will get all that.... eventually.... probably | |
| 18:04:35 | figleaf | win 15 | |
| 18:04:50 | figleaf | ugh | |
| 18:05:20 | leakypipes | SamYaple: ack | |
| 18:08:14 | figleaf | leakypipes: +1 on separate code path | |
| 18:08:39 | figleaf | leakypipes: I'd rather avoid lots of nested logic | |
| 18:08:53 | leakypipes | ya | |
| 18:25:01 | mriedem | melwitt: how required are the dependent os-brick changes for these backports? https://review.openstack.org/#/q/Id188d48609f3d22d14e16c7f6114291d547a8986 | |
| 18:25:19 | mriedem | because we can bump upper-constraints in stable but not global-requirements | |
| 18:27:23 | melwitt | mriedem: without the os-brick changes, the 'nova start' after a compute host reboot would fail because the device won't be found during the detach during re-initialization | |
| 18:27:39 | mriedem | hmm | |
| 18:27:40 | mriedem | well then | |
| 18:28:04 | mriedem | i don't think we can backport those then | |
| 18:28:16 | melwitt | I've requested .z release for os-brick stable/pike and stable/ocata, jungleboyj said he can do that next week | |
| 18:28:19 | mriedem | unless you have some compat code in nova to handle os-brick not having that functionality | |
| 18:29:28 | melwitt | yeah, we can handle the os-brick exit code thing in nova. do you think that's more appropriate for backport land? it would make things easier for me | |
| 18:29:56 | melwitt | I had been thinking we had to go the .z stable release + upper-constraints route | |
| 18:30:44 | mriedem | upper-constraints doesn't mean people are going to use that versoin | |
| 18:30:48 | mriedem | that's what g-r is for | |
| 18:31:22 | mriedem | rhosp can of course require whichever package deps it wants for patch releases, but upstream we generally don't bump the minimum required version of a dependency on stable b/c we have to assume people have already shipped and possibly frozen those deps | |
| 18:32:19 | mriedem | "we can handle the os-brick exit code thing in nova" is probably best, if you can determine the failure and log something like, "make sure you upgrade os-brick to avoid this problem" or whatever | |
| 18:32:29 | melwitt | yeah, I know. I thought it was going to be like, people will get the compatible os-brick because the latest .z release would be picked up automatically. is that not how it works? does it grab the minimum only? | |
| 18:33:08 | mriedem | our CI uses what's in upper-constraints | |
| 18:33:20 | mriedem | we can't assume that's how people deploy though | |
| 18:33:24 | mriedem | at least post-GA | |
| 18:34:10 | melwitt | I see. okay | |
| 18:35:17 | openstackgerrit | sean mooney proposed openstack/os-vif stable/pike: Check if interface belongs to a Linux Bridge before removing https://review.openstack.org/531465 | |
| 18:35:23 | melwitt | the os-brick thing is just that we added an additional exit code for a detach call to ignore, that means "ignore if not found". we can catch that in nova too, from os-brick, and look at the exit code and ignore it if the code == "not found" | |
| 18:35:57 | mriedem | by parsing the message? | |
| 18:36:02 | melwitt | it's just more correct to do the check in os-brick since os-brick is the one making the detach call | |
| 18:36:07 | mriedem | oh | |
| 18:36:07 | melwitt | no | |
| 18:36:19 | mriedem | ok | |
| 18:36:22 | mriedem | well that doesn't seem too terrible | |
| 18:36:31 | melwitt | os-brick will raise ProcessExecutionError and one of the attributes is what exit code it was | |
| 18:36:36 | mriedem | yeah | |
| 18:36:41 | melwitt | yeah, it's not terrible IMHO either | |
| 18:38:57 | pooja | Happy Friday all! I had an upgrade related question for nova database.. is it safe to truncate the nova.reservations table when upgrading the controller? As per my understanding, when there is no instance creation in progress, all rows will always be in soft-deleted state. | |
| 18:40:33 | mriedem | pooja: why not just run the "nova-manage db archive_deleted_rows" CLI? | |
| 18:43:30 | pooja | mriedem: I ran that db command, but it doesn't seem to cleanup reservations table rows | |
| 18:45:26 | mriedem | which release are you on? does the command complete without errors? what value are you using for --max_rows? | |
| 18:45:54 | pooja | I am on Newton release.. Ran the command as - nova-manage --debug --config-file /etc/nova/nova.conf db archive_deleted_rows --max_rows 100 | |
| 18:46:22 | pooja | It didn't throw any errors but I also don't see any logs if it did anything | |
| 18:46:36 | mriedem | --max_rows is likely not a big enough number | |
| 18:47:04 | mriedem | --max_rows is the total number, so if you remove 100 records in the first table processed, you likely are done before getting to the reservations table | |
| 18:47:37 | pooja | ah okay.. I thought that number was per table. Let me try with a higher value | |
| 18:47:57 | mriedem | ildikov: well, running without the pike uca now but n-cpu fails to start because: "libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Permission denied" | |
| 18:48:30 | mriedem | pooja: you can run it with the --verbose option | |
| 18:48:30 | mriedem | https://github.com/openstack/nova/blob/stable/newton/nova/cmd/manage.py#L836 | |
| 18:48:54 | mriedem | pooja: in ocata there is a --until_complete option too https://github.com/openstack/nova/blob/stable/ocata/nova/cmd/manage.py#L665 | |
| 18:49:00 | mriedem | so you don't have to deal with --max_rows | |
| 18:49:15 | mriedem | you could run that in a venv/container, or backport that change to your internal repo | |
| 18:49:34 | pooja | mriedem: Thanks! Running it with max_rows=1000 couple of times seems to have purged the reservations table. Out of curiosity, is truncating the table safe operation during downtime though? | |
| 18:50:06 | ildikov | mriedem: hmm, I saw this error before but I couldn't tell right now whether it was during testing multi-attach or not | |
| 18:50:11 | mriedem | pooja: archive_deleted_rows takes records where deleted != 0 and moves them to the shadow table, then deletes them from the main table | |
| 18:50:21 | mriedem | shadow tables are named shadow_<table_name> | |
| 18:50:26 | mriedem | so shadow_instances, shadow_reservations, etc | |
| 18:50:37 | mriedem | ildikov: i have a feeling it's related to not using pike uca | |
| 18:50:52 | mriedem | pooja: archive deleted records should be safe about any time, but yeah certainly during downtime/maintenance | |
| 18:50:58 | mriedem | *downtime of the control plane | |
| 18:51:15 | ildikov | mriedem: in what sense do you mean? | |
| 18:51:28 | mriedem | ildikov: that's the only thing that's different about this CI run, | |
| 18:51:50 | mriedem | is that it's not using the pike uca, which is the default in devstack - so i'm guessing maybe something changed in devstack and implicitly relied on newer libvirt/qemu, | |
| 18:51:55 | mriedem | and now doesn't work if you're not using that | |
| 18:52:10 | pooja | Got it.. makes sense. It may be a good practice to run archive_deleted_rows periodically as a cronjob with low max_rows count such that there is no visible performance impact then? | |
| 18:52:10 | ildikov | mriedem: I was just wondering what's different in the setup that can lead to a permission denied error | |
| 18:52:12 | mriedem | could be that libvirtd isn't even running | |
| 18:52:36 | mriedem | pooja: i'd ask that in the #openstack-operators channel or mailing list | |
| 18:52:44 | mriedem | to see what other operators are doing | |
| 18:53:11 | pooja | mriedem: Sounds good, will do that. Thanks a lot for your help! | |