| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-05-24 | |||
| 17:10:58 | mriedem | djohnsto: in a meeting now, | |
| 17:11:04 | mriedem | that probably needs to be in the mailing list | |
| 17:11:08 | mriedem | as it's a change in behavior | |
| 17:11:10 | djohnsto | Ok | |
| 17:14:07 | savvas | smcginnis: I removed it but it didn't change anything in terms of the volume create command failing when AZ is not defined | |
| 17:14:43 | smcginnis | savvas: OK, will have to look in to it then. | |
| 17:17:59 | savvas | smcginnis: appreciate it | |
| 17:19:36 | mriedem | savvas: you restarted the services after changing the config right? | |
| 17:23:58 | savvas | mriedem: yes | |
| 17:53:08 | mriedem | anyone remember when the guy in the nova onboarding session asked how he knows if an instance variable is a dict or an object or something else? | |
| 17:53:20 | mriedem | same question, but block_device_mapping | |
| 17:53:40 | mriedem | is a list, is it a single thing? dict or object? if object, which type of object? | |
| 17:53:42 | mriedem | super fun | |
| 17:54:31 | melwitt | I remember | |
| 17:54:46 | artom | mdbooth wrote a thing about that | |
| 17:55:19 | artom | http://lists.openstack.org/pipermail/openstack-dev/2016-June/097529.html | |
| 17:55:39 | melwitt | and yeah. generally I've noticed, things are objects until you get into the virt driver, then a lot of things are dicts | |
| 17:55:46 | artom | The fact that you need a novel to understand wtf is concerning, but at least we now have this novel for posterity | |
| 17:57:07 | mriedem | "as there is also something else typically called 'block_device_mapping' which is not a BlockDeviceMapping object." | |
| 17:57:13 | mriedem | or a BlockDeviceMappingList | |
| 17:57:15 | mriedem | which is the case in places | |
| 17:57:43 | mriedem | sometime the actual variable is bdm passed to a method whose formal parameter name is bdms | |
| 17:59:15 | mriedem | ha, wow | |
| 17:59:26 | artom | No no, I want to help :) | |
| 17:59:29 | mriedem | wait, my wrists or yours? | |
| 18:00:00 | artom | Yes. | |
| 18:00:17 | mriedem | i was specifically looking at _prep_block_device | |
| 18:00:24 | mriedem | which i think takes in a BlockDeviceMappingList | |
| 18:00:48 | openstackgerrit | Lars Kellogg-Stedman proposed openstack/nova master: provide interface-scoped nameserver information https://review.openstack.org/467699 | |
| 18:00:50 | mriedem | and through the magic of driver.get_block_device_info(instance, bdms) converst that list to a list of DriverBlockDevice objects | |
| 18:00:57 | mriedem | which are wrappers around a BlockDeviceMapping objec | |
| 18:00:58 | mriedem | *object | |
| 18:22:44 | tidwellr | I'm probably grasping at straws, but is there a chance anyone here can make sense of this error http://paste.openstack.org/show/610594 seen on stable/mitaka? | |
| 18:23:12 | mriedem | gmann: weren't you talking about something with hide addresses earlier today? ^ | |
| 18:25:02 | mriedem | tidwellr: the instance isn't in the req cache for some reason https://github.com/openstack/nova/blob/stable/mitaka/nova/api/openstack/compute/hide_server_addresses.py#L62 | |
| 18:25:34 | tidwellr | mriedem: that's what it looks like to me, the question is why | |
| 18:25:53 | mriedem | it should be cached here https://github.com/openstack/nova/blob/stable/mitaka/nova/api/openstack/compute/servers.py#L443 | |
| 18:25:55 | mriedem | so not sure | |
| 18:26:14 | mriedem | unless the extension's show() method is processed before the instance is cached? | |
| 18:26:21 | mriedem | sdague: ^ any idea how that craziness works? | |
| 18:26:28 | tidwellr | what's weird is the user that reported this can run operations from the CLI just fine, but has an app that invokes the API directly and that is where we see this | |
| 18:27:17 | tidwellr | I'm wondering if it is related to auth/domains somehow, again I'm grasping at straws here | |
| 18:28:43 | sdague | mriedem: I need you to summarize the question | |
| 18:28:49 | mriedem | sdague: heh, | |
| 18:28:58 | mriedem | how do extensions work in the api :) | |
| 18:29:07 | artom | mriedem, what I'm seeing is that the instance is obtained here: https://github.com/openstack/nova/blob/stable/mitaka/nova/api/openstack/compute/hide_server_addresses.py#L62 | |
| 18:29:24 | mriedem | artom: it has to be None | |
| 18:29:26 | artom | Is that from cache? Or is it missing from the bd | |
| 18:29:32 | mriedem | it's not in the cache | |
| 18:29:39 | mriedem | AttributeError: 'NoneType' object has no attribute 'get' | |
| 18:30:01 | mriedem | sdague: do the servers extensions for show() happen after the actual servers controller is called? | |
| 18:30:22 | sdague | yes, iirc. This is mitaka? | |
| 18:30:23 | mriedem | sdague: i.e. we do GET /servers/{server_id} and store the instance in the request cache here https://github.com/openstack/nova/blob/stable/mitaka/nova/api/openstack/compute/servers.py#L443 | |
| 18:30:26 | mriedem | yeah | |
| 18:30:33 | sdague | that's my recollection | |
| 18:30:43 | sdague | tidwellr: what was going on before the stack trace? | |
| 18:31:23 | mriedem | i see other server extensions like for security groups handle this differently | |
| 18:32:54 | mriedem | every other extension does something like https://github.com/openstack/nova/blob/stable/mitaka/nova/api/openstack/compute/servers.py#L443 | |
| 18:32:56 | mriedem | oops | |
| 18:33:01 | mriedem | req.get_db_instance(server['id']) | |
| 18:33:10 | mriedem | they don't use the id parameter in the 'show' method | |
| 18:33:10 | tidwellr | sdague: I've got a user who is making calls to v2.1/<project ID>/servers/<server ID> | |
| 18:33:25 | tidwellr | in this case, it's a GET | |
| 18:33:27 | sdague | tidwellr: what kind of calls? GET calls? | |
| 18:33:38 | sdague | tidwellr: is it immediately after a POST? | |
| 18:35:07 | tidwellr | sdague: no, we're troublshooting this call against an instance that's been running for a while | |
| 18:35:29 | sdague | tidwellr: do you have any local custom extensions? | |
| 18:35:42 | tidwellr | sdague: no, we don't | |
| 18:35:47 | sdague | hmmm | |
| 18:36:10 | mriedem | https://github.com/openstack/nova/blob/stable/mitaka/nova/api/openstack/wsgi.py#L619 | |
| 18:36:15 | mriedem | what are **action_args here? | |
| 18:37:18 | sdague | that's basically the peek of the actions dict iirc | |
| 18:37:33 | mriedem | but if this is a GET | |
| 18:37:40 | sdague | so it should be nothing | |
| 18:37:46 | mriedem | ok, so that would be the issue | |
| 18:37:55 | sdague | it really shouldn't be | |
| 18:38:13 | mriedem | if id passed to the show() method is None https://github.com/openstack/nova/blob/stable/mitaka/nova/api/openstack/compute/hide_server_addresses.py#L62 | |
| 18:38:42 | mriedem | note that every other show() extension gets the instance from the cache doing something like this https://github.com/openstack/nova/blob/stable/mitaka/nova/api/openstack/compute/config_drive.py#L52 | |
| 18:38:44 | mriedem | from the response object | |
| 18:38:55 | mriedem | resp_obj.obj['servers']['id'] | |
| 18:40:27 | sdague | mriedem: no, that's different | |
| 18:40:45 | sdague | https://github.com/openstack/nova/blob/d3297e2e73a2578ffc2987facf5b894667056f97/nova/api/openstack/compute/config_drive.py#L32 | |
| 18:41:35 | sdague | get_db_instance goes into the request instance cache | |
| 18:41:52 | sdague | that other thing is where it's munging the resp obj before it comes back | |
| 18:42:14 | sdague | tidwellr: is there anything notable before that part of the log? | |
| 18:42:56 | tidwellr | sdague: let me take another look, not that I recall | |
| 18:43:00 | tidwellr | just a sec | |
| 18:43:33 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Provide original fault message when BFV fails https://review.openstack.org/467715 | |
| 18:46:03 | tidwellr | sdague: my logs are pretty quiet leading up this | |
| 18:46:33 | tidwellr | sdague: it only shows up when the tenant makes the call to show the instance | |
| 18:46:51 | mriedem | i don't know what would be different between CLI and a direct curl request to the api | |
| 18:47:16 | sdague | tidwellr: ..... why is the instance UUID uppercased? | |
| 18:47:19 | sdague | 5100C340-8967-4DB1-97F7-BFA1C155F250 | |
| 18:47:20 | tidwellr | the difference I'm thinking is the way the user is getting a token | |
| 18:48:07 | sdague | because, you know, that could actually totally cause this problem if that id is not case matched to the db id | |
| 18:48:12 | melwitt | mriedem: I was thinking I'm gonna just write a functional test that tests the counting across cells for server group members, without trying to do anything more complicated with scheduling. I'm thinking to try to set up two cell databases, put an instance in each that is associated with the same group, and see if the counting function returns the right thing | |
| 18:48:26 | sdague | because... mysql | |
| 18:49:02 | mriedem | melwitt: that's probably a good idea | |
| 18:49:18 | melwitt | cool | |
| 18:49:50 | dansmith | melwitt: not that you asked me, but I agree too :) | |