| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-01-05 | |||
| 18:40:33 | mriedem | pooja: why not just run the "nova-manage db archive_deleted_rows" CLI? | |
| 18:43:30 | pooja | mriedem: I ran that db command, but it doesn't seem to cleanup reservations table rows | |
| 18:45:26 | mriedem | which release are you on? does the command complete without errors? what value are you using for --max_rows? | |
| 18:45:54 | pooja | I am on Newton release.. Ran the command as - nova-manage --debug --config-file /etc/nova/nova.conf db archive_deleted_rows --max_rows 100 | |
| 18:46:22 | pooja | It didn't throw any errors but I also don't see any logs if it did anything | |
| 18:46:36 | mriedem | --max_rows is likely not a big enough number | |
| 18:47:04 | mriedem | --max_rows is the total number, so if you remove 100 records in the first table processed, you likely are done before getting to the reservations table | |
| 18:47:37 | pooja | ah okay.. I thought that number was per table. Let me try with a higher value | |
| 18:47:57 | mriedem | ildikov: well, running without the pike uca now but n-cpu fails to start because: "libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Permission denied" | |
| 18:48:30 | mriedem | pooja: you can run it with the --verbose option | |
| 18:48:30 | mriedem | https://github.com/openstack/nova/blob/stable/newton/nova/cmd/manage.py#L836 | |
| 18:48:54 | mriedem | pooja: in ocata there is a --until_complete option too https://github.com/openstack/nova/blob/stable/ocata/nova/cmd/manage.py#L665 | |
| 18:49:00 | mriedem | so you don't have to deal with --max_rows | |
| 18:49:15 | mriedem | you could run that in a venv/container, or backport that change to your internal repo | |
| 18:49:34 | pooja | mriedem: Thanks! Running it with max_rows=1000 couple of times seems to have purged the reservations table. Out of curiosity, is truncating the table safe operation during downtime though? | |
| 18:50:06 | ildikov | mriedem: hmm, I saw this error before but I couldn't tell right now whether it was during testing multi-attach or not | |
| 18:50:11 | mriedem | pooja: archive_deleted_rows takes records where deleted != 0 and moves them to the shadow table, then deletes them from the main table | |
| 18:50:21 | mriedem | shadow tables are named shadow_<table_name> | |
| 18:50:26 | mriedem | so shadow_instances, shadow_reservations, etc | |
| 18:50:37 | mriedem | ildikov: i have a feeling it's related to not using pike uca | |
| 18:50:52 | mriedem | pooja: archive deleted records should be safe about any time, but yeah certainly during downtime/maintenance | |
| 18:50:58 | mriedem | *downtime of the control plane | |
| 18:51:15 | ildikov | mriedem: in what sense do you mean? | |
| 18:51:28 | mriedem | ildikov: that's the only thing that's different about this CI run, | |
| 18:51:50 | mriedem | is that it's not using the pike uca, which is the default in devstack - so i'm guessing maybe something changed in devstack and implicitly relied on newer libvirt/qemu, | |
| 18:51:55 | mriedem | and now doesn't work if you're not using that | |
| 18:52:10 | pooja | Got it.. makes sense. It may be a good practice to run archive_deleted_rows periodically as a cronjob with low max_rows count such that there is no visible performance impact then? | |
| 18:52:10 | ildikov | mriedem: I was just wondering what's different in the setup that can lead to a permission denied error | |
| 18:52:12 | mriedem | could be that libvirtd isn't even running | |
| 18:52:36 | mriedem | pooja: i'd ask that in the #openstack-operators channel or mailing list | |
| 18:52:44 | mriedem | to see what other operators are doing | |
| 18:53:11 | pooja | mriedem: Sounds good, will do that. Thanks a lot for your help! | |
| 18:53:17 | mriedem | yw | |
| 18:53:44 | mriedem | looks like libvirtd is running http://logs.openstack.org/86/531386/2/check/tempest-full/4eacc3c/controller/logs/libvirt/ | |
| 18:56:17 | mriedem | hmm | |
| 18:56:17 | mriedem | 2018-01-05 16:50:52.125+0000: 11136: error : virSecurityDriverLookup:80 : unsupported configuration: Security driver apparmor not enabled | |
| 18:59:57 | mriedem | hmm https://review.openstack.org/#/c/451492/ | |
| 19:00:14 | SamYaple | ive definetely seen that error | |
| 19:00:15 | SamYaple | alot | |
| 19:00:44 | mriedem | which version? | |
| 19:00:51 | mriedem | or maybe we just don't care about the apparmor error | |
| 19:01:04 | ildikov | there are a couple other errors there too | |
| 19:01:26 | SamYaple | mriedem: primarily on 14.04, but that would have been with UCA libvirt 1.3.1 | |
| 19:01:53 | SamYaple | mriedem: ive seen it on 16.04 too, but only when some old 14.04 configs werent cleaned up properly | |
| 19:01:55 | mriedem | this is xenial so 1.3.1 | |
| 19:01:57 | openstackgerrit | Jay Pipes proposed openstack/nova master: add _has_provider_trees() utility function https://review.openstack.org/531474 | |
| 19:02:11 | SamYaple | i do not see it when running libvirt in containers with UCA libvirt | |
| 19:02:14 | mriedem | ii libvirt-bin 1.3.1-1ubuntu10.15 | |
| 19:02:36 | SamYaple | yep, thats the version ive seen it with, but it was 14.04 with mitaka UCA | |
| 19:03:02 | mriedem | this is just devstack with ENABLE_UBUNTU_CLOUD_ARCHIVE=False | |
| 19:03:40 | SamYaple | now that were talking about this, i dont know if that error broke anything, i just remember seeing it | |
| 19:03:44 | SamYaple | it could be a red herring | |
| 19:06:28 | openstackgerrit | melanie witt proposed openstack/nova master: Add access_url_base to console_auth_tokens table https://review.openstack.org/334614 | |
| 19:06:29 | openstackgerrit | melanie witt proposed openstack/nova master: Optionalize instance_uuid in console_auth_token_get_valid() https://review.openstack.org/481700 | |
| 19:06:29 | openstackgerrit | melanie witt proposed openstack/nova master: Add ConsoleAuthToken object https://review.openstack.org/320063 | |
| 19:06:30 | openstackgerrit | melanie witt proposed openstack/nova master: Add periodic task to clean expired console tokens https://review.openstack.org/325381 | |
| 19:06:30 | openstackgerrit | melanie witt proposed openstack/nova master: Use ConsoleAuthToken object to generate authorizations https://review.openstack.org/325414 | |
| 19:06:31 | openstackgerrit | melanie witt proposed openstack/nova master: Convert websocketproxy to use db for token validation https://review.openstack.org/333990 | |
| 19:06:31 | openstackgerrit | melanie witt proposed openstack/nova master: Add index on token_hash and instance_uuid for console_auth_tokens https://review.openstack.org/531477 | |
| 19:06:49 | clarkb | mriedem: SamYaple the issue with non UCA libvirt was segfaulting and other memory access faults | |
| 19:07:01 | clarkb | when we upgraded to UCA libvirt those went away (not sure if that is helpful) | |
| 19:07:09 | mriedem | clarkb: i'm getting a permission denied error on startup | |
| 19:07:21 | mriedem | connecting to libvirt-sock | |
| 19:07:26 | clarkb | mriedem: thats the group issue I think | |
| 19:07:36 | clarkb | (I thought we handled both possible groups but maybe that is broken?) | |
| 19:07:40 | mriedem | clarkb: right i was looking at this https://review.openstack.org/#/c/451492/15/stackrc | |
| 19:07:48 | SamYaple | clarkb: that helps me, and i think its safe to ignore those apparmor issues fwiw | |
| 19:08:15 | clarkb | SamYaple: we never root caused the memory issues because upstream basically said EOLD and upgrade | |
| 19:08:16 | melwitt | mriedem: when I separate the index add, is it supposed to get its own migration file under migrate_repo/versions/? | |
| 19:08:28 | mriedem | melwitt: yeah it would have to | |
| 19:08:32 | melwitt | oops. okay | |
| 19:08:52 | SamYaple | clarkb: upstream as in libvirt? or ubuntu? because ubuntu should be fixing that for the next 3(?) years or so | |
| 19:08:59 | SamYaple | clarkb: did you pop a bug with canonical? | |
| 19:09:11 | clarkb | SamYaple: upstream as in libvirt, ya we filed a bug against canonical/ubuntu | |
| 19:09:41 | mriedem | hmm, do we not have a worlddump anymore? | |
| 19:09:44 | mriedem | did that dump the groups? | |
| 21:15:48 | dansmith | and I dunno what to do with the randomize thing.. maybe also a flag about whether to do that or not (spread/pack) ? | |
| 21:15:57 | mriedem | dansmith: that's in that change | |
| 21:15:59 | mriedem | so that's already done | |
| 21:16:03 | dansmith | oh, | |
| 21:16:04 | leakypipes | dansmith: that's already there. | |
| 21:16:11 | dansmith | placement has a config not a query param for the randomize? | |
| 21:16:12 | mriedem | i think the idea being, if your limit is smaller, you do random, and if it's bigger, you don't | |
| 21:16:15 | leakypipes | sorry, that's why I was confused :) | |
| 21:16:15 | dansmith | gotcha | |
| 21:16:18 | mriedem | dansmith: correct | |
| 21:16:33 | mriedem | does seem like it should be a per-request on the client side thing | |
| 21:16:34 | dansmith | right I hadn't actually paid attention to that part | |
| 21:16:40 | mriedem | config-driven api behavior | |
| 21:17:03 | dansmith | the randomize? I would have kinda thought so, since cinder and nova might have different requirements for which 1000 to return | |
| 21:17:16 | mriedem | yeah | |
| 21:17:25 | mriedem | welp, i guess that's a future improvement for rocky | |
| 21:17:28 | dansmith | yeah, that's what I had been assuming, but it | |
| 21:17:33 | dansmith | is something we can do later | |
| 21:17:33 | dansmith | yeah | |
| 21:18:21 | mriedem | i wonder if we'd want to make the limit option on the nova side a multiple of the number of instances to build in a given request...or is that getting too cute? | |
| 21:18:55 | dansmith | opens a DoS though | |
| 21:19:05 | mriedem | true | |
| 21:19:24 | dansmith | I would make it static and then if people need cuteness, we can ask for safe suggestions | |
| 21:20:01 | mriedem | alright, now how to track this - could do it against the blueprint if someone is going to work on it now, else i can open a bug since it's kind of a bug on the nova side that we don't limit | |