| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-07-08 | |||
| 03:33:54 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: WIP: Add Cyborg device profile groups to request spec. https://review.opendev.org/631243 | |
| 03:33:55 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: WIP: Get resolved Cyborg ARQs and add PCI BDFs to VM's domain XML. https://review.opendev.org/631245 | |
| 03:43:28 | prometheanfire | I'm guessing volumes showing up as attached in nova but not cinder (and not actually being attached) is a nova problem? | |
| 03:43:34 | prometheanfire | can't 'remove volume' it No volume with a name or ID of '3e92f9b6-1a3b-4a7e-8487-6ff253e888db' exists. | |
| 03:50:11 | prometheanfire | problem starts when attaching it, times out with a 504 :| | |
| 04:22:03 | openstackgerrit | Takashi NATSUME proposed openstack/python-novaclient master: Deprecate cells v1 and extension commands and APIs https://review.opendev.org/669597 | |
| 06:16:30 | openstackgerrit | Arthur Dayne proposed openstack/nova-specs master: Proposal for a safer noVNC console with password authentication https://review.opendev.org/623120 | |
| 06:45:02 | openstackgerrit | Merged openstack/nova master: Clean up test_virtapi https://review.opendev.org/667419 | |
| 07:20:44 | openstackgerrit | wangwei1 proposed openstack/nova master: fix spelling error in nova/api/validation/__init__.py https://review.opendev.org/669244 | |
| 07:22:06 | openstackgerrit | Boxiang Zhu proposed openstack/nova master: Add host and hypervisor_hostname flag to create server https://review.opendev.org/645520 | |
| 08:41:22 | openstackgerrit | Merged openstack/nova master: Add VirtAPI.update_compute_provider_status https://review.opendev.org/668706 | |
| 09:51:20 | openstackgerrit | Brin Zhang proposed openstack/nova master: Specify availability_zone to unshelve https://review.opendev.org/663851 | |
| 09:57:49 | openstackgerrit | Miguel Ángel Herranz Trillo proposed openstack/nova stable/queens: Fix type error on call to mount device https://review.opendev.org/669629 | |
| 10:16:06 | moshele | @sean-k-mooney: hi | |
| 10:37:25 | gokhani | hi team, because of power outage, Most of our compute nodes unexpectedly shut down and now I can not start our instances. Instances Power status is No State. Error log is http://paste.openstack.org/show/754107/. My environment is OpenStack Pike and Instances are on a nfs shared storage. Nova version is 16.1.6.dev2. There are more important instances on my environment. How can I rescue my istances? What would you suggest ? | |
| 11:19:25 | openstackgerrit | Ivaylo Mitev proposed openstack/nova master: VMware VMDK detach: get adapter type from instance VM https://review.opendev.org/653738 | |
| 11:56:19 | openstackgerrit | Merged openstack/nova master: Correct the comment of RequestSpec's network_metadata https://review.opendev.org/667061 | |
| 12:33:08 | openstackgerrit | Brin Zhang proposed openstack/nova master: Specify availability_zone to unshelve https://review.opendev.org/663851 | |
| 12:35:24 | openstackgerrit | Shilpa Devharakar proposed openstack/nova master: Support filtering of hosts by forbidden aggregates https://review.opendev.org/667952 | |
| 12:57:44 | openstackgerrit | Merged openstack/os-resource-classes master: Add Python 3 Train unit tests https://review.opendev.org/669479 | |
| 13:00:04 | openstackgerrit | Merged openstack/os-traits master: Add Python 3 Train unit tests https://review.opendev.org/669480 | |
| 13:25:41 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Fix type error on call to mount device https://review.opendev.org/669664 | |
| 13:41:51 | bauzas | mriedem: around ? | |
| 13:42:01 | bauzas | mriedem: I replied to the thread but I wonder about something | |
| 13:42:06 | mriedem | yes | |
| 13:42:27 | bauzas | in both http://paste.openstack.org/show/734146/ and https://github.com/larsks/os-placement-tools/blob/master/check_placement.py they first ask about all the instances | |
| 13:42:55 | bauzas | and then they loop around allocations by checking some persisted object | |
| 13:43:05 | bauzas | but then, we could have some races | |
| 13:43:20 | bauzas | plus, it could be super huge to get all the instances for CERN eg. | |
| 13:43:36 | bauzas | so, I wonder whether I should call the API for every instance | |
| 13:43:49 | bauzas | erh | |
| 13:43:53 | bauzas | I mean, checking the DB | |
| 13:46:03 | mriedem | i'd think you'd avoid races by checking if an instance has a task_state != None, and if looking at migrations, is the migration in progress | |
| 13:48:09 | mriedem | mnaser's script doesn't loop over instances, | |
| 13:48:27 | mriedem | it gets hypervisors and the servers on those hypervisors and compares to the resource provider inventory/allocation per hypervisor/server | |
| 13:50:13 | mriedem | i'd think it would be easiest to first determine what kinds of problems the audit command is going to try and find | |
| 13:50:29 | mriedem | and then figure out the best way to implement that | |
| 13:51:08 | mriedem | i think we know the two main issues are missing allocations (which heal_allocations should fix) and leaked / orphaned allocations, so i'd start by focusing on the latter issue | |
| 13:55:17 | mriedem | i think we could pretty easily identify leaked allocations by starting from the provider allocations and working backward - looking for servers or migration records with the allocation consumer id (since we don't have the type recorded yet) | |
| 13:59:30 | mriedem | so i'd probably start by getting all hypervisors (nodes/resource providers) and for each hypervisor, get the allocations - similar to what mnaser's script does | |
| 14:00:00 | mriedem | getting the hypervisors + servers in the same call like his script would also tell you if allocations on that provider was a server or not - and if not, it's either leaked or it's a migration allocation | |
| 14:00:09 | mriedem | so no i don't think you need to loop over all instances | |
| 14:00:18 | mriedem | maybe i should just write this thing :) | |
| 14:06:37 | bauzas | mriedem: I'm pretty sure I can do it | |
| 14:16:29 | openstackgerrit | François Palin proposed openstack/nova master: WIP - call volume detach even if terminate_connection fails https://review.opendev.org/669674 | |
| 14:19:01 | kashyap | mriedem: You're right ... indeed I didn't hear any responses on the list so far (http://lists.openstack.org/pipermail/openstack-discuss/2019-July/007521.html) | |
| 14:19:33 | kashyap | alex_xu: Hiya, when you're about, do you have a comment on the above thread? (And the change wich Matt Cced you on, which removes Intel CMT events) | |
| 14:20:10 | efried | kashyap: As predicted, my wife immediately stole the sleep book | |
| 14:20:20 | efried | and it is life-changing | |
| 14:20:41 | kashyap | efried: Hehe. Sharing is caring. I'm sure she will not hide her fascination with the book | |
| 14:20:56 | efried | I don't know why, but stuff I've been telling her for years for some reason is having more of an impact coming from a complete stranger | |
| 14:20:57 | kashyap | efried: Ah, she already finished it? | |
| 14:21:20 | efried | no, hasn't finished yet, but every chapter brings new insight. | |
| 14:21:26 | efried | she has bought a copy for her parents | |
| 14:21:31 | efried | and recommended it to a number of people. | |
| 14:21:36 | kashyap | :-) | |
| 14:21:49 | efried | I got a couple chapters in before she stole it, and I'm definitely sold | |
| 14:21:57 | kashyap | efried: I know, you must've become like a "broken record", and people zone out the moment broken reel start spinning :D | |
| 14:22:06 | kashyap | Sorry, just teasing. | |
| 14:22:35 | efried | though for me it's more fascination with the science than altering lifestyle, which was already pretty close to right sleep-wise. | |
| 14:23:07 | kashyap | Yeah, agreed. | |
| 14:23:46 | kashyap | efried: Also, I'd highly recommend to watch him give a tech talk. He's one of the most effective speakers I've ever seen. | |
| 14:24:01 | efried | noted | |
| 14:24:08 | kashyap | The way he pauses for the words to sink in, diction, is all worth observing. At least to me. | |
| 14:24:11 | kashyap | https://www.youtube.com/watch?v=aXflBZXAucQ | |
| 14:24:42 | kashyap | efried: If you click, look how quickly (within 5 seconds) he escalates the talk. :D | |
| 14:29:27 | prometheanfire | was pointed here for nova adding a cinder volume to a server without (the record/db at least) but not actually adding it to the running VM (client reports a 504 when trying to add the volume) | |
| 14:30:28 | prometheanfire | it shows up in the volumes_attached field, but can't remove any of them as they either do not exist or..... | |
| 14:30:32 | prometheanfire | Invalid volume: Invalid input received: Invalid volume: Unable to detach volume. Volume status must be 'in-use' and attach_status must be 'attached' to detach. (HTTP 400) (Request-ID: req-01d8a3c5-66f4-43a4-bac5-9f1104a292fe) (HTTP 400) (Request-ID: req-358132e4-38a1-493a-b87b-d135791bae2d) | |
| 15:02:46 | efried | It seems straightforward enough, but I'm not sure if there could e.g. be security problems, or if there's another accepted way to do this, etc. | |
| 15:02:46 | efried | https://review.opendev.org/#/c/667975/ | |
| 15:02:46 | efried | kashyap: Would you mind taking a look at this please? | |
| 15:03:09 | kashyap | efried: In a meeting; will look | |
| 15:03:15 | efried | no hurry | |
| 15:03:22 | efried | thanks | |
| 15:09:28 | stephenfin | lyarwood: Could you send this on its merry way? https://review.opendev.org/#/c/667355/ | |
| 15:11:04 | lyarwood | stephenfin: looking | |
| 15:11:48 | lyarwood | stephenfin: sorry I did mean to sort that out last week once the stein change landed | |
| 15:11:52 | lyarwood | stephenfin: done | |
| 15:13:07 | mriedem | gibi: comments in https://review.opendev.org/#/c/669188/ just so i'm sure i'm following it correctly (i know this patch was because i brought it up elsewhere) | |
| 15:13:57 | gibi | mriedem: looking.. | |
| 15:26:43 | gibi | mriedem: replied. I only added some extra explanation but I think we agree. | |
| 15:27:22 | gibi | mriedem: I can re-spin the patch to fix the test redundancies | |
| 15:28:46 | mriedem | yeah just fix the test and i'm +2 | |
| 15:28:57 | gibi | mriedem: OK, I will do it quickly | |
| 15:29:51 | cdent | mriedem, efried : I have a situation where under high load the _update_to_placement call made at the end of instance_claim in the compute manager can fail on ResourceProviderConflict all 4 retry attempts (because of nova-scheduler simultaneously making many allocations to the same resource provider). This fails the instance build even though the instance did build. | |
| 15:30:52 | efried | cdent: That is very interesting. | |
| 15:31:04 | cdent | I'm considering a hack to make instance_claim not call _update_to_placement and just letting the periodic job do that. The reason _update_to_placement is actually talking to placement is because inventory (DISK_GB max_unit) does change after many (most) instance creations | |
| 15:31:30 | cdent | ideally that max_unit thing wouldn't be in place but that's not really any immediate option | |
| 15:31:36 | efried | I'm guessing backing off the retry interval wouldn't be of help, because the problem is the timing between the GET and the PUT(s). | |
| 15:31:49 | sean-k-mooney | cdent: were we not looking at disabling the periodic job at one point | |
| 15:31:51 | efried | wait, what? | |
| 15:32:00 | efried | max_unit changes? | |
| 15:32:24 | cdent | to account for multiple different source of one disk_gb in a cluster | |
| 15:32:26 | mriedem | cdent: is the vcenter driver reporting different max_unit DISK_GB from update_provider_tree during the claim? | |
| 15:32:28 | cdent | being presented as one disk | |
| 15:32:47 | sean-k-mooney | cdent: you mean for things like ceph | |
| 15:32:59 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Remove assumption of http error if consumer not exists https://review.opendev.org/669188 | |
| 15:33:00 | mriedem | this is likely not libvirt/ceph | |
| 15:33:10 | mriedem | it's vcenter with shared storage provider or something? | |