| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-01-05 | |||
| 18:31:22 | mriedem | rhosp can of course require whichever package deps it wants for patch releases, but upstream we generally don't bump the minimum required version of a dependency on stable b/c we have to assume people have already shipped and possibly frozen those deps | |
| 18:32:19 | mriedem | "we can handle the os-brick exit code thing in nova" is probably best, if you can determine the failure and log something like, "make sure you upgrade os-brick to avoid this problem" or whatever | |
| 18:32:29 | melwitt | yeah, I know. I thought it was going to be like, people will get the compatible os-brick because the latest .z release would be picked up automatically. is that not how it works? does it grab the minimum only? | |
| 18:33:08 | mriedem | our CI uses what's in upper-constraints | |
| 18:33:20 | mriedem | we can't assume that's how people deploy though | |
| 18:33:24 | mriedem | at least post-GA | |
| 18:34:10 | melwitt | I see. okay | |
| 18:35:17 | openstackgerrit | sean mooney proposed openstack/os-vif stable/pike: Check if interface belongs to a Linux Bridge before removing https://review.openstack.org/531465 | |
| 18:35:23 | melwitt | the os-brick thing is just that we added an additional exit code for a detach call to ignore, that means "ignore if not found". we can catch that in nova too, from os-brick, and look at the exit code and ignore it if the code == "not found" | |
| 18:35:57 | mriedem | by parsing the message? | |
| 18:36:02 | melwitt | it's just more correct to do the check in os-brick since os-brick is the one making the detach call | |
| 18:36:07 | melwitt | no | |
| 18:36:07 | mriedem | oh | |
| 18:36:19 | mriedem | ok | |
| 18:36:22 | mriedem | well that doesn't seem too terrible | |
| 18:36:31 | melwitt | os-brick will raise ProcessExecutionError and one of the attributes is what exit code it was | |
| 18:36:36 | mriedem | yeah | |
| 18:36:41 | melwitt | yeah, it's not terrible IMHO either | |
| 18:38:57 | pooja | Happy Friday all! I had an upgrade related question for nova database.. is it safe to truncate the nova.reservations table when upgrading the controller? As per my understanding, when there is no instance creation in progress, all rows will always be in soft-deleted state. | |
| 18:40:33 | mriedem | pooja: why not just run the "nova-manage db archive_deleted_rows" CLI? | |
| 18:43:30 | pooja | mriedem: I ran that db command, but it doesn't seem to cleanup reservations table rows | |
| 18:45:26 | mriedem | which release are you on? does the command complete without errors? what value are you using for --max_rows? | |
| 18:45:54 | pooja | I am on Newton release.. Ran the command as - nova-manage --debug --config-file /etc/nova/nova.conf db archive_deleted_rows --max_rows 100 | |
| 18:46:22 | pooja | It didn't throw any errors but I also don't see any logs if it did anything | |
| 18:46:36 | mriedem | --max_rows is likely not a big enough number | |
| 18:47:04 | mriedem | --max_rows is the total number, so if you remove 100 records in the first table processed, you likely are done before getting to the reservations table | |
| 18:47:37 | pooja | ah okay.. I thought that number was per table. Let me try with a higher value | |
| 18:47:57 | mriedem | ildikov: well, running without the pike uca now but n-cpu fails to start because: "libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Permission denied" | |
| 18:48:30 | mriedem | https://github.com/openstack/nova/blob/stable/newton/nova/cmd/manage.py#L836 | |
| 18:48:30 | mriedem | pooja: you can run it with the --verbose option | |
| 18:48:54 | mriedem | pooja: in ocata there is a --until_complete option too https://github.com/openstack/nova/blob/stable/ocata/nova/cmd/manage.py#L665 | |
| 18:49:00 | mriedem | so you don't have to deal with --max_rows | |
| 18:49:15 | mriedem | you could run that in a venv/container, or backport that change to your internal repo | |
| 18:49:34 | pooja | mriedem: Thanks! Running it with max_rows=1000 couple of times seems to have purged the reservations table. Out of curiosity, is truncating the table safe operation during downtime though? | |
| 18:50:06 | ildikov | mriedem: hmm, I saw this error before but I couldn't tell right now whether it was during testing multi-attach or not | |
| 18:50:11 | mriedem | pooja: archive_deleted_rows takes records where deleted != 0 and moves them to the shadow table, then deletes them from the main table | |
| 18:50:21 | mriedem | shadow tables are named shadow_<table_name> | |
| 18:50:26 | mriedem | so shadow_instances, shadow_reservations, etc | |
| 18:50:37 | mriedem | ildikov: i have a feeling it's related to not using pike uca | |
| 18:50:52 | mriedem | pooja: archive deleted records should be safe about any time, but yeah certainly during downtime/maintenance | |
| 18:50:58 | mriedem | *downtime of the control plane | |
| 18:51:15 | ildikov | mriedem: in what sense do you mean? | |
| 18:51:28 | mriedem | ildikov: that's the only thing that's different about this CI run, | |
| 18:51:50 | mriedem | is that it's not using the pike uca, which is the default in devstack - so i'm guessing maybe something changed in devstack and implicitly relied on newer libvirt/qemu, | |
| 18:51:55 | mriedem | and now doesn't work if you're not using that | |
| 18:52:10 | ildikov | mriedem: I was just wondering what's different in the setup that can lead to a permission denied error | |
| 18:52:10 | pooja | Got it.. makes sense. It may be a good practice to run archive_deleted_rows periodically as a cronjob with low max_rows count such that there is no visible performance impact then? | |
| 18:52:12 | mriedem | could be that libvirtd isn't even running | |
| 18:52:36 | mriedem | pooja: i'd ask that in the #openstack-operators channel or mailing list | |
| 18:52:44 | mriedem | to see what other operators are doing | |
| 18:53:11 | pooja | mriedem: Sounds good, will do that. Thanks a lot for your help! | |
| 18:53:17 | mriedem | yw | |
| 18:53:44 | mriedem | looks like libvirtd is running http://logs.openstack.org/86/531386/2/check/tempest-full/4eacc3c/controller/logs/libvirt/ | |
| 18:56:17 | mriedem | 2018-01-05 16:50:52.125+0000: 11136: error : virSecurityDriverLookup:80 : unsupported configuration: Security driver apparmor not enabled | |
| 18:56:17 | mriedem | hmm | |
| 18:59:57 | mriedem | hmm https://review.openstack.org/#/c/451492/ | |
| 19:00:14 | SamYaple | ive definetely seen that error | |
| 19:00:15 | SamYaple | alot | |
| 19:00:44 | mriedem | which version? | |
| 19:00:51 | mriedem | or maybe we just don't care about the apparmor error | |
| 19:01:04 | ildikov | there are a couple other errors there too | |
| 19:01:26 | SamYaple | mriedem: primarily on 14.04, but that would have been with UCA libvirt 1.3.1 | |
| 19:01:53 | SamYaple | mriedem: ive seen it on 16.04 too, but only when some old 14.04 configs werent cleaned up properly | |
| 19:01:55 | mriedem | this is xenial so 1.3.1 | |
| 19:01:57 | openstackgerrit | Jay Pipes proposed openstack/nova master: add _has_provider_trees() utility function https://review.openstack.org/531474 | |
| 19:02:11 | SamYaple | i do not see it when running libvirt in containers with UCA libvirt | |
| 19:02:14 | mriedem | ii libvirt-bin 1.3.1-1ubuntu10.15 | |
| 19:02:36 | SamYaple | yep, thats the version ive seen it with, but it was 14.04 with mitaka UCA | |
| 19:03:02 | mriedem | this is just devstack with ENABLE_UBUNTU_CLOUD_ARCHIVE=False | |
| 19:03:40 | SamYaple | now that were talking about this, i dont know if that error broke anything, i just remember seeing it | |
| 19:03:44 | SamYaple | it could be a red herring | |
| 19:06:28 | openstackgerrit | melanie witt proposed openstack/nova master: Add access_url_base to console_auth_tokens table https://review.openstack.org/334614 | |
| 19:06:29 | openstackgerrit | melanie witt proposed openstack/nova master: Add ConsoleAuthToken object https://review.openstack.org/320063 | |
| 19:06:29 | openstackgerrit | melanie witt proposed openstack/nova master: Optionalize instance_uuid in console_auth_token_get_valid() https://review.openstack.org/481700 | |
| 19:06:30 | openstackgerrit | melanie witt proposed openstack/nova master: Use ConsoleAuthToken object to generate authorizations https://review.openstack.org/325414 | |
| 19:06:30 | openstackgerrit | melanie witt proposed openstack/nova master: Add periodic task to clean expired console tokens https://review.openstack.org/325381 | |
| 19:06:31 | openstackgerrit | melanie witt proposed openstack/nova master: Add index on token_hash and instance_uuid for console_auth_tokens https://review.openstack.org/531477 | |
| 19:06:31 | openstackgerrit | melanie witt proposed openstack/nova master: Convert websocketproxy to use db for token validation https://review.openstack.org/333990 | |
| 19:06:49 | clarkb | mriedem: SamYaple the issue with non UCA libvirt was segfaulting and other memory access faults | |
| 19:07:01 | clarkb | when we upgraded to UCA libvirt those went away (not sure if that is helpful) | |
| 19:07:09 | mriedem | clarkb: i'm getting a permission denied error on startup | |
| 19:07:21 | mriedem | connecting to libvirt-sock | |
| 19:07:26 | clarkb | mriedem: thats the group issue I think | |
| 19:07:36 | clarkb | (I thought we handled both possible groups but maybe that is broken?) | |
| 19:07:40 | mriedem | clarkb: right i was looking at this https://review.openstack.org/#/c/451492/15/stackrc | |
| 19:07:48 | SamYaple | clarkb: that helps me, and i think its safe to ignore those apparmor issues fwiw | |
| 19:08:15 | clarkb | SamYaple: we never root caused the memory issues because upstream basically said EOLD and upgrade | |
| 19:08:16 | melwitt | mriedem: when I separate the index add, is it supposed to get its own migration file under migrate_repo/versions/? | |
| 19:08:28 | mriedem | melwitt: yeah it would have to | |
| 19:08:32 | melwitt | oops. okay | |
| 19:08:52 | SamYaple | clarkb: upstream as in libvirt? or ubuntu? because ubuntu should be fixing that for the next 3(?) years or so | |
| 19:08:59 | SamYaple | clarkb: did you pop a bug with canonical? | |
| 19:09:11 | clarkb | SamYaple: upstream as in libvirt, ya we filed a bug against canonical/ubuntu | |
| 19:09:41 | mriedem | hmm, do we not have a worlddump anymore? | |
| 19:09:44 | mriedem | did that dump the groups? | |
| 21:15:48 | dansmith | and I dunno what to do with the randomize thing.. maybe also a flag about whether to do that or not (spread/pack) ? | |
| 21:15:57 | mriedem | dansmith: that's in that change | |
| 21:15:59 | mriedem | so that's already done | |
| 21:16:03 | dansmith | oh, | |
| 21:16:04 | leakypipes | dansmith: that's already there. | |