| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-01-05 | |||
| 18:02:48 | SamYaple | i just want to use the native qemu interface more directly. live differential backups. add/remove hardware. better metrics. that kind of thing | |
| 18:02:59 | SamYaple | but libvirt will get all that.... eventually.... probably | |
| 18:04:35 | figleaf | win 15 | |
| 18:04:50 | figleaf | ugh | |
| 18:05:20 | leakypipes | SamYaple: ack | |
| 18:08:14 | figleaf | leakypipes: +1 on separate code path | |
| 18:08:39 | figleaf | leakypipes: I'd rather avoid lots of nested logic | |
| 18:08:53 | leakypipes | ya | |
| 18:25:01 | mriedem | melwitt: how required are the dependent os-brick changes for these backports? https://review.openstack.org/#/q/Id188d48609f3d22d14e16c7f6114291d547a8986 | |
| 18:25:19 | mriedem | because we can bump upper-constraints in stable but not global-requirements | |
| 18:27:23 | melwitt | mriedem: without the os-brick changes, the 'nova start' after a compute host reboot would fail because the device won't be found during the detach during re-initialization | |
| 18:27:39 | mriedem | hmm | |
| 18:27:40 | mriedem | well then | |
| 18:28:04 | mriedem | i don't think we can backport those then | |
| 18:28:16 | melwitt | I've requested .z release for os-brick stable/pike and stable/ocata, jungleboyj said he can do that next week | |
| 18:28:19 | mriedem | unless you have some compat code in nova to handle os-brick not having that functionality | |
| 18:29:28 | melwitt | yeah, we can handle the os-brick exit code thing in nova. do you think that's more appropriate for backport land? it would make things easier for me | |
| 18:29:56 | melwitt | I had been thinking we had to go the .z stable release + upper-constraints route | |
| 18:30:44 | mriedem | upper-constraints doesn't mean people are going to use that versoin | |
| 18:30:48 | mriedem | that's what g-r is for | |
| 18:31:22 | mriedem | rhosp can of course require whichever package deps it wants for patch releases, but upstream we generally don't bump the minimum required version of a dependency on stable b/c we have to assume people have already shipped and possibly frozen those deps | |
| 18:32:19 | mriedem | "we can handle the os-brick exit code thing in nova" is probably best, if you can determine the failure and log something like, "make sure you upgrade os-brick to avoid this problem" or whatever | |
| 18:32:29 | melwitt | yeah, I know. I thought it was going to be like, people will get the compatible os-brick because the latest .z release would be picked up automatically. is that not how it works? does it grab the minimum only? | |
| 18:33:08 | mriedem | our CI uses what's in upper-constraints | |
| 18:33:20 | mriedem | we can't assume that's how people deploy though | |
| 18:33:24 | mriedem | at least post-GA | |
| 18:34:10 | melwitt | I see. okay | |
| 18:35:17 | openstackgerrit | sean mooney proposed openstack/os-vif stable/pike: Check if interface belongs to a Linux Bridge before removing https://review.openstack.org/531465 | |
| 18:35:23 | melwitt | the os-brick thing is just that we added an additional exit code for a detach call to ignore, that means "ignore if not found". we can catch that in nova too, from os-brick, and look at the exit code and ignore it if the code == "not found" | |
| 18:35:57 | mriedem | by parsing the message? | |
| 18:36:02 | melwitt | it's just more correct to do the check in os-brick since os-brick is the one making the detach call | |
| 18:36:07 | melwitt | no | |
| 18:36:07 | mriedem | oh | |
| 18:36:19 | mriedem | ok | |
| 18:36:22 | mriedem | well that doesn't seem too terrible | |
| 18:36:31 | melwitt | os-brick will raise ProcessExecutionError and one of the attributes is what exit code it was | |
| 18:36:36 | mriedem | yeah | |
| 18:36:41 | melwitt | yeah, it's not terrible IMHO either | |
| 18:38:57 | pooja | Happy Friday all! I had an upgrade related question for nova database.. is it safe to truncate the nova.reservations table when upgrading the controller? As per my understanding, when there is no instance creation in progress, all rows will always be in soft-deleted state. | |
| 18:40:33 | mriedem | pooja: why not just run the "nova-manage db archive_deleted_rows" CLI? | |
| 18:43:30 | pooja | mriedem: I ran that db command, but it doesn't seem to cleanup reservations table rows | |
| 18:45:26 | mriedem | which release are you on? does the command complete without errors? what value are you using for --max_rows? | |
| 18:45:54 | pooja | I am on Newton release.. Ran the command as - nova-manage --debug --config-file /etc/nova/nova.conf db archive_deleted_rows --max_rows 100 | |
| 18:46:22 | pooja | It didn't throw any errors but I also don't see any logs if it did anything | |
| 18:46:36 | mriedem | --max_rows is likely not a big enough number | |
| 18:47:04 | mriedem | --max_rows is the total number, so if you remove 100 records in the first table processed, you likely are done before getting to the reservations table | |
| 18:47:37 | pooja | ah okay.. I thought that number was per table. Let me try with a higher value | |
| 18:47:57 | mriedem | ildikov: well, running without the pike uca now but n-cpu fails to start because: "libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Permission denied" | |
| 18:48:30 | mriedem | https://github.com/openstack/nova/blob/stable/newton/nova/cmd/manage.py#L836 | |
| 18:48:30 | mriedem | pooja: you can run it with the --verbose option | |
| 18:48:54 | mriedem | pooja: in ocata there is a --until_complete option too https://github.com/openstack/nova/blob/stable/ocata/nova/cmd/manage.py#L665 | |
| 18:49:00 | mriedem | so you don't have to deal with --max_rows | |
| 18:49:15 | mriedem | you could run that in a venv/container, or backport that change to your internal repo | |
| 18:49:34 | pooja | mriedem: Thanks! Running it with max_rows=1000 couple of times seems to have purged the reservations table. Out of curiosity, is truncating the table safe operation during downtime though? | |
| 18:50:06 | ildikov | mriedem: hmm, I saw this error before but I couldn't tell right now whether it was during testing multi-attach or not | |
| 18:50:11 | mriedem | pooja: archive_deleted_rows takes records where deleted != 0 and moves them to the shadow table, then deletes them from the main table | |
| 18:50:21 | mriedem | shadow tables are named shadow_<table_name> | |
| 18:50:26 | mriedem | so shadow_instances, shadow_reservations, etc | |
| 18:50:37 | mriedem | ildikov: i have a feeling it's related to not using pike uca | |
| 18:50:52 | mriedem | pooja: archive deleted records should be safe about any time, but yeah certainly during downtime/maintenance | |
| 18:50:58 | mriedem | *downtime of the control plane | |
| 18:51:15 | ildikov | mriedem: in what sense do you mean? | |
| 18:51:28 | mriedem | ildikov: that's the only thing that's different about this CI run, | |
| 18:51:50 | mriedem | is that it's not using the pike uca, which is the default in devstack - so i'm guessing maybe something changed in devstack and implicitly relied on newer libvirt/qemu, | |
| 18:51:55 | mriedem | and now doesn't work if you're not using that | |
| 18:52:10 | ildikov | mriedem: I was just wondering what's different in the setup that can lead to a permission denied error | |
| 18:52:10 | pooja | Got it.. makes sense. It may be a good practice to run archive_deleted_rows periodically as a cronjob with low max_rows count such that there is no visible performance impact then? | |
| 18:52:12 | mriedem | could be that libvirtd isn't even running | |
| 18:52:36 | mriedem | pooja: i'd ask that in the #openstack-operators channel or mailing list | |
| 18:52:44 | mriedem | to see what other operators are doing | |
| 18:53:11 | pooja | mriedem: Sounds good, will do that. Thanks a lot for your help! | |
| 18:53:17 | mriedem | yw | |
| 18:53:44 | mriedem | looks like libvirtd is running http://logs.openstack.org/86/531386/2/check/tempest-full/4eacc3c/controller/logs/libvirt/ | |
| 18:56:17 | mriedem | 2018-01-05 16:50:52.125+0000: 11136: error : virSecurityDriverLookup:80 : unsupported configuration: Security driver apparmor not enabled | |
| 18:56:17 | mriedem | hmm | |
| 18:59:57 | mriedem | hmm https://review.openstack.org/#/c/451492/ | |
| 19:00:14 | SamYaple | ive definetely seen that error | |
| 19:00:15 | SamYaple | alot | |
| 19:00:44 | mriedem | which version? | |
| 19:00:51 | mriedem | or maybe we just don't care about the apparmor error | |
| 19:01:04 | ildikov | there are a couple other errors there too | |
| 19:01:26 | SamYaple | mriedem: primarily on 14.04, but that would have been with UCA libvirt 1.3.1 | |
| 19:01:53 | SamYaple | mriedem: ive seen it on 16.04 too, but only when some old 14.04 configs werent cleaned up properly | |
| 19:01:55 | mriedem | this is xenial so 1.3.1 | |
| 19:01:57 | openstackgerrit | Jay Pipes proposed openstack/nova master: add _has_provider_trees() utility function https://review.openstack.org/531474 | |
| 19:02:11 | SamYaple | i do not see it when running libvirt in containers with UCA libvirt | |
| 19:02:14 | mriedem | ii libvirt-bin 1.3.1-1ubuntu10.15 | |
| 19:02:36 | SamYaple | yep, thats the version ive seen it with, but it was 14.04 with mitaka UCA | |
| 19:03:02 | mriedem | this is just devstack with ENABLE_UBUNTU_CLOUD_ARCHIVE=False | |
| 19:03:40 | SamYaple | now that were talking about this, i dont know if that error broke anything, i just remember seeing it | |
| 19:03:44 | SamYaple | it could be a red herring | |
| 19:06:28 | openstackgerrit | melanie witt proposed openstack/nova master: Add access_url_base to console_auth_tokens table https://review.openstack.org/334614 | |
| 19:06:29 | openstackgerrit | melanie witt proposed openstack/nova master: Add ConsoleAuthToken object https://review.openstack.org/320063 | |
| 19:06:29 | openstackgerrit | melanie witt proposed openstack/nova master: Optionalize instance_uuid in console_auth_token_get_valid() https://review.openstack.org/481700 | |
| 19:06:30 | openstackgerrit | melanie witt proposed openstack/nova master: Use ConsoleAuthToken object to generate authorizations https://review.openstack.org/325414 | |
| 19:06:30 | openstackgerrit | melanie witt proposed openstack/nova master: Add periodic task to clean expired console tokens https://review.openstack.org/325381 | |
| 19:06:31 | openstackgerrit | melanie witt proposed openstack/nova master: Add index on token_hash and instance_uuid for console_auth_tokens https://review.openstack.org/531477 | |
| 19:06:31 | openstackgerrit | melanie witt proposed openstack/nova master: Convert websocketproxy to use db for token validation https://review.openstack.org/333990 | |
| 19:06:49 | clarkb | mriedem: SamYaple the issue with non UCA libvirt was segfaulting and other memory access faults | |
| 19:07:01 | clarkb | when we upgraded to UCA libvirt those went away (not sure if that is helpful) | |